The questions and answers in this document are CC0 and in the public domain.
Audrey Tang first of all is known as the creator and developer of Pugs, the Perl 6 User’s Golfing System, an implementation of Perl 6 in Haskell, which started on 1 February 2005 and was the most actively developing and the most complete implementation at that time.
Background
You started learning Perl when you were 12?
I encountered Perl from Randal Schwartz’s Llama Book when I was 13. I followed closely the development of the OnePerl effort during the 5.004-5.005 development, helped establishing communities such as the Usenet group tw.bbs.comp.lang.perl, and learned much from the nascent CPAN community.
Was it your first language?
My first computer languages, when I was 8, were GW-BASIC and Logo (a LISP variant). I’ve been programming in various languages (LPC for LPMud, etc.) before the Web arrived in 1994.
Apart from Pugs, you’ve got lots of modules in Perl 5 published on CPAN since 2001. What was your pre-Perl 6 life connected with Perl?
After the web arrived, Perl pretty much covered all my programming needs for the next 10 years. During 1999-2000, I migrated from EFNet to IRCNet and the new Rhizomatic server (aka MAGNet, later irc.perl.org), where I learned of the global Perl Monger initiative and the lighter-hearted Perl Golf initiatives.
At that time we were translating the Llama Book’s 3rd edition to Traditional Chinese, so it seemed natural to form Taipei.pm and connect with the wider CPAN community. I principally worked on CPANPLUS, the CPAN smoke-test framework, Module::Signature, as well as the PAR/pp packaging tool.
When and why did you decide to work on Perl 6?
I attended OSCON 2003 to give a tutorial on PAR.pm — though I spent most of the flight preparing a lightning talk, which received a very warm welcome from the community. The last lightning talk in the same session was Allison’s Restaurant, with these particularly inspiring words:
You know, if one person, just one person, does it, they may think he’s really sick and lock him up. And if two people do it, in harmony, they may think they’re both nuts and lock ’em both up. And if three people do it! Can you imagine three people walkin’ in, singin’ a bar of “Perl 6 Development” and walkin’ out? They may think it’s an organization! And can you imagine fifty people a day? I said FIFTY people a day . . . walkin’ in, singin’ a bar of “Perl 6 Development” and walkin’ out? And friends, they may think it’s a MOVEMENT, and that’s just what it is: THE ANTI-FIGHTIN’, CODE CLEANIN’, FEATURE ADDIN’, PARROT LOVIN’, PERL 6 DEVELOPMENT MOVEMENT! . . . and all you gotta do to join is to sing it the next time it comes around.
That’s when I decided to work on Perl 6.
Before Pugs, you actively contributed to Parrot. What exactly did you do there?
I participated with Dan Kogai on the perl6-internals list on Parrot’s internationalization infrastructure.
Other than that I’ve just tried out new builds occasionally and reported build issues; I wasn’t actively involved with Parrot during the early years when Simon Cozens and Dan Sugalski led the effort.
When you started it, did you think about creating both a compiler and interpreter?
Yes. In 2005-02-26 the Pugs Apocrypha already talked about compiling to Haskell, Parrot and Perl 5.
-Ofun
You are the author of the ‘-Ofun’ meme, or “Optimizing for fun”. What does it mean for you and what was it for the rest of the team?
There’s a slide deck explaining what it meant for me:
For community participants, it meant a culture of warm reception, regular updates, highlighting details, personal feedback, forming a ladder of skills and helping each other.
For software hackers, the culture of -Ofun focuses on embracing distributed versioning, encouraging forks, preferring ideas over consensus, hugging trolls, and sketching ideas with code.
What could be the path of the project, would you choose the opposite motto, like “no premature optimization”?
-Ofun really means the same thing as “no premature optimization”, because premature optimization would restrict the possibilities of exploration and so decrease the fun.
In the case of the logical opposite, i.e. had I chose “optimize for boredom” or “optimize for misery”, I think we’ll reach the local maximum rather quickly, ending up with a rather deterministic path — namely racing to the ⊥. 🙂
But seriously, compiling and testing earlier Pugs versions took literary hours. What did you do to speed that up?
We waited for Moore’s law to catch up, which worked pretty well. Also, we refactored the large, monolithic modules into smaller pieces, which made incremental compilation and testing quite a bit faster.
For the Pugs releases, you used approximation of π as version numbers. Is it also part of the -Ofun thing?
Indeed. It’s doubling the fun of the TeX versioning system.
The TeX’s versioning assumes that no big changes will be made in the future versions. With the Perl 6 implementation, you had a number of milestones:
- 6.0: Initial release
- 6.2: Basic IO and control flow elements; mutable variables; assignment
- 6.28: Classes and traits
- 6.283: Rules and Grammars
- 6.2831: Type system and linking
- 6.28318: Macros
- 6.283185: Port Pugs to Perl 6, if needed
How close were you following this list?
Pretty close. Pugs had implemented the first two milestones according to the Synopses, and we were well on the way toward the third milestone. The early implementers of the 6.28 milestone eventually initiated the Moose project, bringing the Perl 6 object system to Perl 5 and JavaScript communities.
As for the next three milestones: While Pugs did help starting design conversations around those aspects, the Synopses for grammars, type and macro systems took much longer to crystallize, with help from the succeeding implementers of Perlito, Niecza and Rakudo.
Haskell
Quoting your post announcing Pugs it was “an attempt at writing a Perl6 interpreter in Haskell”. Why Haskell?
I was learning Haskell at that time in the irc.freenode.net haskell channel:
Pugs was my second Haskell program, initially done as an exercise for the book “Types and Programming Languages” (page 82 from the slides above).
Haskell is a purely functional programming language, while Perl 6 is not. Which parts of implementation became easier because of this choice, and which did not fit with Haskell very well?
It was helpful in that it’s much easier to reason about Perl 6 semantics, making explicit all ambiguities in the specification, because Haskell did not prefer one semantic over another.
As for shortcomings, my 2010 writeup contains a list. The Haskell community was still young when we started Pugs, and I’m glad to see the cross-fertilization brought both communities forward.
You often mentioned monads in your blog posts; what did you do with them in Pugs?
The interpreter is a monad of this type:
runEvalT :: ContT (EvalResult Val) (ReaderT Env SIO) (EvalResult a) -> Eval a -- data SIO a = MkSTM !(STM a) | MkIO !(IO a) | MkSIO !a
This expresses explicitly the continuation semantics, result type, run-time lexical environment, and distinguishes between states of software-transactional memory, of input-output effects, as well as pure code.
The parser is another monad that embeds the interpreter monad, allowing for BEGIN blocks to modify parsing logic during compilation.
Did Haskell nature help implementing the async and start keywords and all the features for parallel an concurrent things in Perl 6?
Certainly. When Perl 6 was exploring designs around parallelism and concurrency, the Haskell community was also figuring out operational semantics of the underlying abstractions, such as threads, events, delimited continuation and so on. It was a good fit to bring the cutting-edge development to test whether the cutting-edge design was sound.
What about Perl 6’s object oriented features, including method resolution order?
Tie-breaking of multi-methods, as well as its interaction with junctions, were among the first things Pugs helped to clarify in Perl 6 design. We also influenced Perl 5’s adoption of pluggable method-resolution order: The core module mro.pm references Pugs as its inspiration, thanks to Stevan Little who did the initial prototyping on C3.
A quote from you: «A 20-minutes hack had paid off handsomely: In 33 lines, we have a fast, pure Haskell implementation of basic Perl 6 Rules support.» That sounds amazing, can you explain how that is possible and how does Haskell (or, apparently, some libraries) deal internally with Perl grammars and rules?
That references Text.Parser.Rule.parseGrammar, a port of Patrick Michaud’s PGE to Haskell’s Parser Arrows. Internally, the arrow abstraction made effects accumulated during parsing much more explicit. Later on, Adrian D. Thurston (independently) fully explored the idea of parsing engines with built-in revert/undo rules, resulting in the Colm programming language.
You also contributed to Haskell itself. Namely, you worked within the Haskell’ (Haskell Prime) project to modernize the Haskell standard. What was your role there and did your work on Pugs changed your vision of Haskell and how it should develop? Or, in fewer words, did Perl 6 influence Haskell?
I worked on Unicode related language proposals such as SourceEncodingDetection; I reviewed and deliberated on other proposals as well. Pugs was an early adopter of GADTs, Delimited Continuations, QuasiQuotation etc, and we provided feedback on GHC to make these features more solid.
Our primary influence on Haskell, though, was drawing people to learn Haskell amongst a massively online collaborative community, instead of its original “Powered by PhDs” academic setting. People who encountered Haskell through Pugs — such as Edward Kmett — would go on to vitalize the Haskell community with the same spirit of intensively collaborative development.
In the same year 2005, you started learning Erlang. Do you think it was possible to build a Perl 6 compiler in that language? Would it be different from the Haskell implementation?
Certainly possible. The main difference is that Haskell — by virtue of making all effects explicit — helped to clarify a set of Perl 6 semantic that’s “native” to Perl 6, which makes it unique among other functional languages at that time.
If not Haskell or Erlang, what other functional languages could be candidates for making a compiler or interpreter of Perl 6?
OCaml would have been a reasonable choice as well, with its emphasis on runtime performance and strong multi-stage programming support.
Among Perl 6’s contemporaries, Scala, Clojure and C# 3.0+ (which Niecza builds on) are all excellent functional languages for Perl 6 implementers to build on.
Compiling a project in Haskell not only required patience but also prerequisites like GHC, the Haskell compiler. It might stop those who wanted to barely play with Perl 6. What did you recommend to such an audience? [maybe the Perl 6 test server, or deb-like packages, or Perl 6 IRC bot, etc.]
Yes: I would recommend the Perl 6 test servers, deb-like packages such as the Haskell Platform and MinGHC, and the Perl 6 IRC bot. (I love self-answering questions! :100: )
The Team
After couple of weeks after Pugs’ launch, there were half a dozen contributors, and in a year — over 200. How did it happen that the project in a language quite different from Perl (I mean Haskell, not Perl 6 :-)) attracted so many people?
There was already significant community interest in Perl 6, and Pugs benefited from the community’s generosity. For example, Scott Walters, who wrote the “Perl 6 Now” book in 2004, kindly donated the Perl 6 code snippets into the public domain, which became some of the earliest materials in Pugs repository.
Actually, relatively few people worked on the Haskell codebase. Due to a strong test-first culture since the inception, the majority of people on IRC contributed by writing Perl 6 code — examples, tests, modules — so the language barrier was not a major deterrence.
How many active contributors were there?
There were 131 active committers during the first year and 395 in total listed in the perl6/mu repository.
How many of them stayed long?
Many comitters either remained in the #perl6 channel or subscribed to perl6-related news and blogs, but the degree of involvement is difficult to quantify.
Nevertheless, as there are more committers to Perl 6 development after 2007 than during 2005-2006, it’s clear that the Perl 6 community has successfully bootstrapped ourselves.
You are known as a project coordinator who easily give out commit bits. What did you do when someone committed something completely different from your vision of the project development?
Usually I would change my vision accordingly — and deliberate with the language designers to change their vision accordingly. In that regard, the IRC channel was instrumental in blending people’s visions together.
Was there any kind of a policy behind that?
There were only (ever improving) procedures and no top-down policies.
Did you have any rules for developers?
The only requirement is that you know how to be nice to all kinds of people (and, some years later, butterflies), or are learning to be nice, or at least want to learn to be nice. As Larry put it:
You don’t yet want to be good, but perhaps you want to want to. We may be able to help with that too. If the acceleration stays positive, eventually the velocity and position will come around too.
How did you resolve conflicts, both code-wise and people-wise?
We followed the Path of Git:
- fetch new facts
- merge with understanding
- commit to action
- push to the world!
Was it difficult to manage such a big team?
I don’t know — I never tried to manage. 🙂
Or it was a chaotic process which you directed to go right way?
Language designers are, by nature, social interaction designers, because languages are ways people interact with each other. Instead of directing, I simply declared — as observed by _why — “there are no wrong ways in this world”, which defined a particular kind of space that allowed for all kinds of concurrent processes.
Presumably, not every Pugs team member was familiar with Haskell? Was it easy for them to work on the project?
As mentioned above, a majority of people on IRC contributed by writing Perl 6 code.
For people interested in picking up Haskell, we made the READTHEM and READTOO files for exactly that purpose. Learning Haskell wasn’t particularly easy — people already versed in imperative programming had to unlearn many habits — but we did our best to make it enjoyable and properly victualed.
Can you name a few team members who did the most significant contribution and tell what they did?
Pugs was thoroughly an ensemble project. If I name the ones I remember along with their significant contributions, the answer to this question would be longer than the rest of this interview combined!
I would say, though, that the ensemble wouldn’t have emerged so swiftly, if not for the existing online community infrastructures in the form of use Perl;, Perl Monks, Perl Mailing Lists, FreeNode IRC and OpenFoundry Project Hosting — these spaces, with their underlying technologies and operators, made this kind of collaboration possible in the first place.
The Process and the Internals
What is the general approach to write a compiler or interpreter in Haskell?
Please refer to Stephen Diehl’s excellent tutorials on implementing imperative and functional languages in Haskell. Pugs initially started with the functional path but quickly changed to follow the imperative path.
Are there any analogues of standard compiler compiler tools like lex and yacc, or bison?
All of them — The compiler tools wiki page has a full list.
What is Parsec, the Haskell library, for?
It is a simple, well documented monadic parser combinator library for fast parsers with good error messages. The “combinator” part means it’s reasonably easy to implement the core Perl 6 feature of changing the grammar on-the-fly during parsing.
One of the ambitious goals of Perl 6 roadmap was to bootstrap it in itself, thus having a compiler written in Perl 6. So far there is a NQP (Not Quite Perl) subset of Perl 6, which is used in Rakudo. Back then, what was your vision of Perl 6 bootstraping with Pugs?
It was the same vision — The only way to begin bootstrapping is by first implementing some parts in other languages, then rewrite those parts in Perl 6. As described in the Pugs apocrypha:
In a bootstrapping process, there are often many bottlenecks, which prevent people from working on things that depend on them. For example, one cannot easily write unit tests and standard libraries for Perl 6 without a working Perl 6 implementation, or work on an AST evaluator without an AST interface. Pugs solves such deadlocks by providing ready substitutes at various levels of the process.
Still, there was some internal representation, the PIL (Pugs Intermediate Language). What kind of a language is that?
It is more of a readable serialization of the Pugs internal AST structure — it’s not a human-writable language.
Are there any relations with PIR, the Parrot’s Intermediate Representation?
Yes, PIL was specifically created to work out semantic mappings between the Perl 6 language and the Parrot virtual machine.
Quite soon after Pugs was launched, it was able to produce code for different backends like Perl 5, JavaScript, and even IMCC (Intermediate Code Compiler) for Parrot. Did this variety bring any help (like finding bugs in implementation, or the language design, or such)?
Certainly it did.
Wasn’t it a waste of resources, which left less time for the Pugs progress?
Well, the fallacy of this scarcity-based assumption was pointed out early on by chromatic on Perl Monks.
We cherished the diversity of contributors who brought their expertise on different backends, as their cross-fertilization made more contributions possible, ultimately accelerating the development beyond what is possible for a monoculture.
Similar thing about Perl 5 compatibility. In the early days, Perl 6 was considered as a language that should be compatible with Perl 5, which is now not the case at all. Was Perl 5 compatibility an issue?
Hmm, is it not the case at all now? Perl 5 compatibility — both at the source level and at embedded interpreter level — is still desirable, and the two approaches are still useful parts of Rakudo Perl.
Although having useful Perl 5 modules, like DBI.pm, available in a program in Perl 6 can be very useful, was it really necessary?
It was instrumental in working out Perl 6’s way of interfacing with foreign languages — starting with the least foreign language (Perl 5) was considerably easier than other languages!
There were a number of hackathons, either dedicated purely for Pugs, or broader ones for Perl 6. How many of them did you attend?
Was it useful? Were there new people who wanted to help with Pugs or Perl 6 development?
Yes — a lot!
More importantly, it gave us perspectives from people who were previously not involved with Perl 6, which let the design team evaluate the ongoing work from the user’s point of view.
How did you choose what to implement next?
We started with a roadmap, which prompted people to write tests pertaining the current progress. Later on, the specs and TODO tests drove much of the direction of implementation.
Did you always follow the TDD principles before writing the code?
Yes if it’s the plural “you” — that is, the community hivemind followed that principle, not me personally. 🙂
Can you explain how do you handle the eval keyword?
We always bundle the Perl 6 compiler within the runtime, so eval is in implemented in 14 lines:
- Try parsing the program and throw an exception if it did not parse
- Enter a new lexical block, inheriting the upper block’s lexicals
- Run INIT block within the eval string
- Start walking through the parsed AST and apply their effects
- Catch exceptions and assign it into $!
Parrot
Parrot was the first virtual machine for Perl 6, and you were among the Parrot developers. Why did you start a separate project?
Parrot was the lower-level virtual machine and runtime; Pugs was the higher-level parser and compiler. The two projects complement each other and intends to meet at the middle — which we eventually did.
Did earlier Pugs versions need Parrot for installation and execution?
The Parrot inter-operability component was optional; so was the Perl5 runtime for loading existing CPAN modules.
What’s the story about licensing the Pugs source directory inside Parrot?
Pugs was among the very first adopters of the revised Artistic License (version 2.0 beta5) as of Feburary 2005. At that time, Parrot was under the dual license of Artistic License (version 1) or GPL (version 2). Parrot eventually upgraded to Artistic License version 2 on June 2007.
Because the two Artistic Licenses are compatible, there were no licensing issues of including Pugs source code inside Parrot.
There were, however, issues of including Pugs source code inside other Haskell libraries and distributions, which are usually available under MIT/BSD style licenses.
To encourage two-way sharing with the Haskell community, I have switched the Haskell part of Pugs, the src/ directory, to the MIT license, which contained chromatic’s contributions. Because chromatic was not comfortable with sharing that Haskell code under the MIT license — which was entirely understandable — he asked me to remove that part of code, and I obliged.
That incident has affected me deeply. Later on, when the Creative Commons project introduced CC0 “No Rights Reserved” in 2009, I have switched all my software work (and other kinds of work) to it, as to remain maximally compatible with other contributors and stakeholders.
The choice and use of a register-based virtual machine for Perl 6 was initially supported by the statement that they are well described in literature. You removed that passage from Parrot’s documentation. Why?
I edited the text to say “compared to stack-based virtual machines, register-based virtual machines takes fewer operations to perform typical high-level language tasks”, because it is easier to quantify that way. For example, when Dalvik VM was introduced in 2007, it averaged ~30% fewer instructions than JVM when interpreting the same Java bytecode.
The initial Parrot documentation alluded to existing research literature on optimizing for hardware registers. In practice, x86-64 had only 16 general-purpose registers — while Parrot and Dalvik supports essentially unlimited amount of registers.
Since there is no obvious one-to-one mapping from hardware-register-based algorithms to software-register-based ones, it’s harder to quantify that aspect as an advantage over stack-based virtual machines.
Were there any literature or research papers available for creating compilers in Haskell?
Yes. In addition to this list of literature on the Haskell Wiki, Oleg Kiselyov’s tagless-final approach is highly instructive as well.
Language Design
While implementing Perl 6, you might have found incompatibilities in the original design documents. How did you improve them?
By getting language designers on IRC and deliberate over the specification documents along with implementors.
Did you offer changing bits of Perl 6’s design in order to make it easy to implement (in Pugs at least)?
Quite the opposite — Aside from resolving ambiguities and contradictions in the spec, most Pugs-driven design changes were invariably making things easier for users and more difficult for implementers.
The idea is that a language’s users far outnumber its implementers, so the utilitarian trade-off makes sense. Consequently, Perl 6 implementers rejoiced in vicarious suffering, and had appropriate amounts of fun.
How clear were Apocalypses and Synopses?
The prose was excellent. Clarity was never the main issue; the main shortcomings were of coherency, consistency, and completeness — all to be discovered by a running implementation.
Eventually Synopses (annotated with tests) becomes the Spec, making it much easier to iterate with an ever-growing community.
What was the main source of the language description for you?
The perl6-language mailing list, and then the perl6 IRC channel.
Did you ever refer to initial RFCs?
Occasionally, often by way of the Apocalypses, to understand the motivating problems.
Pugs Apocrypha — what are they?
Initially written in English and Chinese, the first (and only) apocrypha was a statement of purpose of Pugs, including a snapshot of the Perl 6 community’s efforts and progress at that time.
Influence
Pugs was the leading project and the most complete Perl 6 implementation in its time. Some features that were tested in Perl 6 were later implemented in Perl 5 too. Did Pugs achieve its initial goal in sense that it was a ‘golfing system’?
If, through our work, Perl 6 has become more expressive over time — reducing (key)strokes, increasing (round) tuits — then Pugs would have fulfilled its initial goal. And I think it did!
How strong did Pugs influence both Perl 5 and Perl 6?
The main influence was that we helped fostering a supportive community so people working on Perl 6 can feel comfortable forking Perl 5. Conversely, people working on Perl 5 also felt more comfortable merging good ideas from Perl 6.
The Pugs Test Suite grew into the official Perl 6 test suite and even was integrated into the Synopses later. How did this happen?
The Perl 6 design team helped keeping the test suite synchronized with the spec changes since the beginning of Pugs. During early 2006, Larry joined the IRC channel and I started working on the spec — since then, modification to the language spec were done in lockstep with tests.
In August 2006, Yichun Zhang improved the continuous integration system, displaying tests inline with the spec, as detailed in this writeup. This makes the tests part of the spec, not particular to Pugs.
In January 2008, Larry implemented a fudge program that adapts the test suite to work on new implementations such as SMOP and Rakudo.
In February 2015, the community declared the tests at perl6/roast as the actual specification, with the Synopses at perl6/specs now relegated as “speculations” that document the experience from the implementations.
In 2006, the Pugs project was stalled. Can you explain why it happened?
Sure — I was physiologically and psychologically unable to continue developing the Haskell codebase (Pugs.hs). As during 2006 our collective focus have shifted to run Perl 6 on Perl 5, there was not much point in further development of Pugs.hs, either.
If you continued working on Pugs after 2006, where could it be now?
My main work on Pugs at 2006 was around Moose and Perlito, so I imagine I’ll help bringing more of Perl 6 semantics to run on Perl 5 faster. However, since the lambdamoose team has done such an amazing job toward that vision, I’m sure my absence had no significant impact.
What is lambdamoose (and lambdacamel)?
Lambda is the common insignia shared by functional programming communities, and camel was Perl’s original mascot, so a lambdacamel is someone who identifies with both communities.
Later on, as Moose brought Perl 6’s object model to Perl 5, we coined the new term lambdamoose for someone who identifies with Perl 5, Perl 6, as well as the larger functional programming community.
Were there any chances to continue working on Pugs after a pause?
For me personally, as far as I could recall — and for the foreseeable future — the task that Pugs set out to do was essentially done. As I wrote in 2010:
The Pugs.hs project exists mainly for historical/archival purposes, not for active development. (Forks are, of course, very much welcome.)
What did happen with the team after you stopped actively working on Pugs?
They lived long, and prospered; please refer to Carl Mäsak’s excellent write-up of the Perl 6 efforts that followed, as well as this collection of Moose articles during 2007-2011.
Similar to how Pugs appeared after Parrot, there was KindaPerl6, or MiniPerl6, or Perlito, appeared out of the Pugs repository. What were the reasons of that?
Each project started for an unique reason — you’d need to ask each initiator individually. 🙂
Sharing a repository was a practical way to re-use the existing infrastructure and made it easier to coordinate changes together.
Did you occasionally see, say at conferences, any products from the Pugs Store?
This shirt, mostly:
Do you follow current Perl 6 development?
Yes.
What do you think of Larry’s announcement about his intentions to make Perl 6.0 production ready by Christmas 2015?
Mu!
In 2005, you published an imaginary Perl 6 timeline curve. Can you extend it to the future? What is the Perl 6’s future?
Perl 6’s future points toward infinite community goodwill:
One thought on “Interview with Audrey Tang”