Carl Mäsak is an application developer for Perl 6. He is the number one Perl 6 bug reporter, the author of November, one of the first real web applications written in Perl 6.
October
You started following the Perl 6 development in 2003–2004, a few years after the Perl 6 project was announced. What was so special about that that attracted your attention for years?
Actually, it was Perl that attracted me. I started using it more and more in my projects at home and at university. Started really liking the language. Back in 2003 we were already at Perl 5.8, I guess — though I won’t swear I was running that recent a version.
Anyway, I was alternating between many Perl-related things. Writing small scripts. Checking out CPAN. Reading stuff like “How to tell if you’re a Perl beginner, a Perl wizard, or a Perl guru”. And, of course, reading the Apocalypses. Apocalypse 12 just came out as I was turning really interested. And then of course Pugs happened, which made everything much more concrete. Suddenly you could actually play around with this new language.
At that time Pugs was the main and only implementation of the language. Can you tell about your experience with it?
There was something magical about the Pugs development process. Never before or since have I seen a project more deserving of the description “agile”. Things were changing and growing on a daily basis, the lead developer Audrey wrote daily updates on use.perl.org (now defunct), and there was a constant flurry of discussion and development on the then-new #perl6 channel. This was around 2005 and 2006. Audrey was extremely prolific, sometimes code in what seemed like negative time. Someone would go “hey, there’s a bug…”, Audrey would go “wait for it”, and seconds later an svn commit with a fix popped up, and you’d wonder whether the speed of light was just exceeded somewhere along the way. Or she’d go to sleep and the commits would keep on coming.
With the kind of extreme velocity that Pugs had, it was impossible not to feel some kind of… disappointment… when switching over to Rakudo. At the time (2007), Rakudo was far behind Pugs in many regards, but also a slower-moving project. In my longing for the return of the golden days of Pugs I remember also feeling that Rakudo was somehow more full of bugs than Pugs. That may not actually be true, but it felt that way. Could be it was partly true, partly an artifact of the Parrot platform, and partly the lower velocity of the project as such. Oh, and I should add that nowadays, making quick changes to Rakudo is a breeze, and sometimes easier than it ever was in Pugs. (Because much of the Rakudo code is in Perl 6.)
Even though Pugs is now a dormant project, it made at least three lasting impressions that are with us even today. The first is the spectest suite, which specifies what Perl 6 is and what a Perl 6 implementation is expected to do. The second is testing culture itself. Not a Perl invention as such, but the Pugs project led the way a lot, and now it’s very much a part of CPAN culture and Perl 6 culture alike. The third is Moose, which was born on the #perl6 channel. Since that time Moose and the Perl 6 object system have cross-pollinated several times, so it’s a bit pointless to ask which one is based on which.
You became famous (for the first time) after you submitted dozens of bug reports to the Perl 6 request tracker (rt.perl.org). What was your motivation?
It was kind of silly. My first RT submission was an update to the STATUS file of the Rakudo project, back in May 2008. As I started using Rakudo more, I found all these little things that made Rakudo just fall over and die. Really simple things, things you would never find in Perl 5 because it’s so mature. Like you’d run a program that was just “$_”, and it would throw an error. Or you’d run just “$” and it would hang! Or you’d try to inherit from something that didn’t exist, and you’d get a Parrot NPE (Null-Pointer Exception). Most of those early findings were simply because I was the first one trying them out, and so it made sense that I send them in as I found them. By July 2008 I was averaging two tickets a day.
How did people working on Perl 6 react?
I remember Patrick Michaud saying something like this at YAPC::EU 2008: “At first I was kind of annoyed at all these tickets coming in, most of them describing issues we were already aware of and just hadn’t got to yet. But then I realized it’s kind of nice to have them in RT!” I remember him saying that, because I felt good about not being a complete annoyance to one of the big names in the Perl 6 world! ☺
It quickly turned into a bit of a gimmick. It was possible to keep up the tempo of about one RT ticket a day for two-three years at least. So almost every day people would see “masak submits rakudobug” on the channel. I tried to submit “streamlined” bug reports, with close to minimal character count while still describing the necessary details. I managed to keep at it, because my brain doesn’t consider it work to open up a new email, paste something into it, edit a little, and click send. Each little such contribution is just four-five minutes of the day.
If you go out in the forests or any other natural ecosystem, there will be plants that turn sunlight into chemical structure, animals that eat the plants, animals that eat other animals, and finally the “decomposers”, fungi and bacteria, that take dead organic material and break it down to allow it to re-enter the cycle and be useful again for the plants and animals. Being a decomposer is not very prestigious, but it’s necessary for the sustainability of the ecosystem. I’ve always thought of my rakudobug-submitting habit as acting as a decomposer: digesting some discrepancy and spewing it out into RT where it can be of use for the higher flora and fauna in the Perl 6 world. There are many other jobs like that in an open-source project, by the way: keeping text files or wikis up-to-date, doing minor code fixes, refactoring, cage cleaning. Relatively small tasks each on their own, but important for long-term survival of the project.
Did you try fixing the bugs you discovered?
I did, but after having just described myself as a fungus, I hardly need to add that it didn’t happen all that often! ☺
I remember attempting some of the problems. Usually I’d get stuck, and then Patrick or Jonathan would come along days or weeks later and fix the thing, and then I’d pore over the patch (at that time, usually PIR code) and try to sort out how it was done.
I fix more of them nowadays, but it’s still a small percentage. The exceptions when I do dive in it’s because I see an easy fix or something I can contribute, so that my time is better spent in Rakudo internals than out in Perl 6 application space.
A number of bug reports were about things, which worked differently from what Apocalypses meant. Or were those primarily real software bugs?
Oh man, there are so many types of bugs. That’s the nice thing. We already covered “this perfectly reasonable code blows up”, and the one you just mentioned might be either “spec says this, Rakudo does something else” or just “not yet implemented”. Both of which are perfectly fine bug reports.
A slightly different category would be “I tried this slightly odd thing, and Rakudo died in a fire”. A very common one, that happens because I do real stuff with Perl 6, is “I refactored A into B, and it stopped working”. (Which shouldn’t happen with refactors.) Also, occasionally “something got fixed in Rakudo, which made something break in my program”!
A special circle in hell is reserved for bugs that involve variable lookup and scoping. Oy! If you’ve never experienced your language runtime looking up a variable for you and coming back with clearly the wrong value, consider yourself lucky!
We have a reference Perl 6 parser called STD. (And any puns you can think up about that name, #perl6 has probably heard before.) Sometimes a bug report would just be “STD can parse this, but Rakudo can’t”.
Though occasionally it’s the other way. One of my all-time favorites was finding out that regex literals can be delimited not just with slashes, various bracketing characters (as in Perl 5), various Unicode bracketing characters (as in Perl 6 spec), but also by a Unicode snowman (☃) and a Unicode comet (☄). To be perfectly clear, those two are not mirror-image bracketing characters. So that was a bug I could submit. I never did find out how the parser managed to accept those two as delimiters. (Patrick’s reaction was something like “what are you doing to my grammar engine!?”.)
After a while, the Rakudo project also started to consider insufficiently clear error messages to be reportable bugs. I really like that sentiment, and more projects would do well to adopt it. Rakudo often has extremely nice error messages these days, thanks to this incessant polishing. If an error message could be improved, it’s RT’d as being “Less Than Awesome”.
November
In 2008 in Copenhagen, you became famous for the second time, when during the YAPC::Europe conference together with Johan Viklund you demonstrated ‘November’, the wiki engine written in Perl 6. How difficult was it to create a real project in the language, which was not fully ready for production?
It was a challenge. Or a hundred of them, in sequence. Many of the bugs submitted came out of the (then secret) November project. Because we were working in secret, I had to scrub the rakudobugs of all traces of wiki work. Some of my bug streamlining habits come from that.
Here’s one obstacle I remember in particular. We wanted to show blue links for pages that exist and red links for pages that weren’t written yet. A page would be represented by a file somewhere on disk. So we had to go check if the file existed. One problem: Rakudo didn’t have a filetest operator for testing for existence. Bummer. So, what can we do? After some thinking, we realized that opening the file in a try block and then setting a flag did the trick. Not pretty, but it worked.
November was basically built entirely out of workarounds like that. For example, November also contains a minimal, horrible partial re-implementation of CPAN’s CGI module. It was ugly but exhilarating to be doing something with Perl 6 at such an early stage.
After that conference, more people joined November development. What was done during that phase?
The wiki engine got a few new features, and quite some time was spent making it faster. I remember horrible response times like 17s if you didn’t precompile the modules. It went down to something like 3s if you did precompile. Eventually we managed to lower it even further.
Mostly though, November slowly went into some kind of maintenance mode as interest in it waned. Don’t tell anyone, but lately it’s been kind of a wacky idea of mine to go up and “re-implement” all of November in a 5-minute lightning talk. It could be done fairly nicely, because nowadays we have DBIish and Bailador and other modules that do the heavy lifting for us.
For November, you also used ‘feather’, the dedicated server for Perl 6 development. What was it and what for?
It was a server that Juerd Waalboer graciously made available for the Perl 6 community. The most common use through the years was probably so that people had a place to host their screen sessions for IRC. But it was also a convenient playground for people who wanted to build the latest Pugs or Rakudo, and play around with modules. For a while, we hosted November from feather.
You initiated the creation of yet another Perl 6 compiler, Yapsi. What was the goal of that project?
Yapsi was many things. It was an in-joke making fun of many things in the Rakudo world. Whereas the author of a Rakudo release announcement would be “happy to announce” the release of a new version of Rakudo, the Yapsi announcement would be heralded with “a certain amusement” or with “undue sprightliness”. The improvements from version to version in Yapsi were always quite modest, but they were introduced with great fanfare. Subroutines were introduced as having been invented by the Yapsi team. Where Rakudo at that time generated code in a format called PIR, Yapsi generated something called SIC. Rakudo’s AST format was named PAST, so of course Yapsi’s had to be named FUTURE.
It was also a little bit of a reaction to some features in Rakudo that were taking a long time. Chiefly the long-standing promise that Perl 6 should be able to parse Perl 6 programs and do interesting things with them. I figured if we started from a compiler that was already fully bootstrapping and written completely in Perl 6 from the get-go, this would be an easier goal to achieve. Well, turns out it was still quite an ambitious goal, and Yapsi never got that far before it stalled. (But the ideas of Perl 6 manipulating Perl 6 are still very much alive and well. On my blog I’m exploring how we can do this using macros and ASTs.)
The third thing that Yapsi was for was learning about parsing, compiling and runtimes. This is probably its lasting legacy. As I was tinkering around with the macro ideas, I realized I needed to build a toy language and compiler (called “007”), and thanks to the Yapsi experience, this was fairly straightforward work.
Having this experience with creating compilers, do you think that 15 years is not that long for such a big thing as a Perl 6 compiler?
Yes, I think it takes a long time. I believe I know why, but I still struggle a bit to put it into words.
On some level it’s because it is a big ambitious project and that takes time to get right. I mean, on some level Perl 6 has seen actual compiler releases at least all the way back to Pugs, arguably further than that. But that’s not what people mean when they ask whether Perl 6 is ready. They mean something like, reasonable feature parity with Perl 5, and compelling reasons to switch.
Not everyone has that threshold for what constitutes “ready for use”. The Perl 6 community is made up of people with a lower threshold than that. For me, Perl 6 became a language I would use regularly already in 2008. It’s been a bumpy ride, but less and less so, and every month that goes by, one has to make fewer excuses about remaining rough corners. This is why you will find some people on forums asserting that Perl 6 has been ready for ages. It depends on one’s requirements.
On another level it’s because we’re taking our time and not wanting to botch up the process. “Good, fast, cheap — pick any two”. Perl 6 is a volunteer effort with a focus on quality, and so it takes time. Both Pugs and Rakudo were instrumental in showing the way, putting actual Perl 6 semantics into the hands of users. In many cases, entire subsystems of Rakudo had to be rewritten because enough feedback needed to be absorbed back in from userland.
I don’t want to make excuses for Perl 6 being a 15-year project. That is a long time by any measure. But I’m heartened by the fact that Perl 6 still makes many people excited, and for the right reasons: it is all the things we like about Perl, but updated for modern needs and ready to take on the 21st century with full vigor. And we haven’t just been sitting around during these years, but trying out Perl 6, reconsidering it, polishing it. I’m glad I’m using the Perl 6 of 2015, not the Perl 6 of 2005.
Here and there, a number of Perl 6 hackathons take place. You participate in some of them. Are they a good place to move Perl 6 further?
Well, it’s kind of weird. Most of us are technologists, and we rely heavily on things like email and Skype and IRC. We’re all probably in the 90th percentile of taking advantage of that kind of tool. I guess as open-source developers we’re kind of automatically also telecommuters.
But there’s something special about meeting in person that all these technologies can’t really replace. Once or twice a year it’s very valuable to sit down together in the same room and hash things out. There’s an atmosphere and intensity in those meetings that we never quite achieve on an IRC channel. And some of that atmosphere and intensity lingers for a while, and helps gel us together as a group. When people are just words on your laptop screen, it’s easy to misinterpret someone, or escalate into a flame war. (“Someone is wrong on the Internet”, right?) But if you just sat next to them on a hackathon couch, and you remember them as a three-dimensional person, then it’s much easier to agree online as well.
December
One of the keys to Perl 5’s success is CPAN. For Perl 6 we have none yet. In Perl 6, the philosophy of referencing the modules is way more complex (for example, you may ask to load a module of a specific version or author). Do you think it is worth doing?
It’s important to look at the goal here: allowing one part of an application to depend on module Foo by one author, while another part depends on Foo by another author. It provides insulation in a big project, against unrelated dependencies treading on each others’ feet.
Is it worth doing? I don’t know, actually. There are many important issues here and they are worth addressing and I’m glad Perl 6 is addressing them. On the other hand, there are many ways to go around solving this problem. The Node community has much less of an issue here in the first place, because imports are strictly lexical. In Perl (5 and 6), imports happen on the package level, and so there is a real risk of collisions.
I don’t have any answers, and I’m mostly watching the module/versioning work from the sidelines. I’m not even saying that lexical imports would be better for Perl, because many programmers enjoy package scoping. Given that, I’m glad we’re exploring this solution space.
There’s a list of modules published on perl6.org together with some installation tools. You stood behind ‘proto’, what is that?
It was a very early package manager for Perl 6. Created in 2009, by 2010 it had already been superseded by a better project which punningly called itself ‘neutro’. And one year after that we got ‘panda’, the current solution.
Now that sounds like a crazy piece of churn. And it was. When you create a language ecosystem, you’re going to need a package manager. You won’t get it completely right from the start, and so you will need to keep evolving it. My strength lies more in initial ideas than in carrying a project all the way, so I’m glad tadzik (author of both neutro and panda) stepped in and did the rewrites. The panda installer now serves us very well, much better than proto ever did.
You’ve been playing with Perl 6 for years. How quickly do you think a new programmer (with or without a Perl 5 background) can learn it?
I think an interested programmer coming from a Perl 5 background still has an easier job. We do get them now and then on the #perl6 channel. It’s still a process and it’s still some work, of course, but you can basically proceed by reading stuff and learning what’s different in Perl 6. That migration path has been pretty good for years.
People are actively working on writing documentation for the other group, people without a Perl 5 background. That one’s a harder nut to crack, because there’s not as much common background we can assume. There’s a documentation effort at doc.perl6.org that’s coming along fine. And at some point we’ll see a book, too, but I don’t dare go on record with a time estimate. A bit like Douglas Adams, the Perl 6 community often likes deadlines because of the whooshing sound they make as they go by. ☺
Did you use Perl 6 in other projects?
Yes. Looking back at my Github activity over the years, it seems I went on to implement two board games, a blog engine, a grammar engine, two compilers, two old-school adventure games, a personal finance management system, too many talk presentation systems, various IRC bots, a diagramming tool, a build tool, a module installer, various mathematical solvers, and a TDD automation tool. And that’s not counting various modules that ended up in the ecosystem.
Some of the above projects were inward-facing tools for Perl 6 developers. (We usually refer to it as “dogfooding”, as in “not just producing dog food, but eating it too”.) Most of them were not. I like to think of my role as waking up every morning and going “hm, we haven’t tried to do X in Perl 6 yet”, with X varying from day to day. Boldly going where no-one has gone before is a great way to find new bugs, too.
Tell us about those games.
I have a soft spot for games. One of them is a 3-D connection game, where if you can’t get past your opponent on the board itself, you can start building upwards and make connections above the opponent. It’s called “Druid” because of the way the board ends up looking a bit like Stonehenge. People who have been at my home have often ended up playing it with me, using LEGO pieces.
The other is a family card game from the time of WWII and the Warsaw uprising. In English it goes by the name “Little Animal Farm”, the rules simple enough for this to be a game for the whole family, but challenging enough for there to be a fair element of strategy to the game. Do you invest in a guard dog in case the fox shows up? Do you breed small animals like rabbits or bigger animals like horses and cows? The first player with one of each animal wins the game.
How do people react when you talk about a Perl 6 project at conferences? Do they consider things you show as scientific research, or a pet project for fun, or they see the potential that Perl 6 has?
I don’t know the answer to that. Maybe all of the above? All I can say is that when I give talks, I do try to give them a practical bent that showcases why Perl 6 is interesting in the real world. The extent to which I succeed with that varies, but that’s life.
Oh, I ought to mention that I’ve even used Perl 6 to deliver talks. The PDFs that resulted were very nice, but unfortunately too big to practically upload. Not really Perl 6’s fault in that case, and more related to how I glued together the slides in the PDF. But still, a very down-to-earth use of Perl 6 in practice. I fully expect to deliver all my talks from a Perl 6 stack at some point.
Beyond the talks, you popularize Perl 6 with your blog, which you started about the same time as the November project was launched. How do you manage writing new posts for years?
This can probably be traced back to mst, and his Perl Iron Man challenge. It was something like, everyone blog at least once a week. The one who can keep it up longest wins. Before that, I had blogged a few times, but not regularly. For over a year, I think, I blogged at least once a week. I don’t anymore, but the challenge made me realize that I have things to write about, it doesn’t have to be all that complicated, the perfect is the enemy of the good, etc. Also, blogging is a nice way to organize one’s thoughts, and it usually creates a nice little swirl of activity around the post.
Oh, I should mention that since use.perl.org went dormant in 2010, my blog sits on top of a static web page generator written in Perl 6. Yes, that’s 250 lines of Perl 6 code powering the blog. I told myself that this blog would be at least a 10-year thing. Guess I’m halfway towards that goal. So far I’m very pleased with how smooth it’s been.
I have a couple of long-term plans with the blog engine still. One milestone would be to remove the Perl 5 dependency it currently has (Mark Pilgrim’s original Markdown parser). This is a case where Perl 6 ought to be able to shine, since it does grammars and parsing well. But so far we haven’t had a good enough Markdown parser to swap in. I also want to implement comments, something that’s mildly challenging to get right on a site that renders statically. The thing that holds me back there a bit is how little I’ve missed having comments on the blog.
When Perl 6 is production ready, how intensively are you going to use it for your day job?
I’m one of the people who aren’t waiting around for someone else to declare Perl 6 production-ready, and so I already use Perl 6 at my day job. This comes fairly naturally since (a) I like Perl 6 and know it well, and (b) it feels good to use Perl 6 for things that are directly related to making money for our company.
Besides having Perl 6 in a number of strategic places in our toolchains (for building courses and presentations), we also sometimes use Perl 6 for rapid prototyping. Jonathan and I were at a client where we managed to get a working prototype going in just a few days, thanks to Perl 6 grammars. Then, after amazing the client with the live demo, we clapped our hands together and said “right, so now we just rewrite the same idea in Java in five times the amount of code”. ☺ More seriously, prototypes are still underused in the industry, and we like to use them whenever we can. Perl 6 is already a really good fit for this.