Interview with Flávio Glock

Flávio Glock is the creator of Perlito, the Perl 6 and Perl 5 compiler, and a v6 module, the one compiling Perl 6 to Perl 5 in Perl 5.

Background

First of all, can you please clarify the names? There are MiniPerl6, KindaPerl6, and Perlito, and a few others, which seem to refer to the same project and its forks.

These are different projects. The whole list is:

PIL-Run was a project to use Perl 5 to execute Perl 6 code. Pugs compiles Perl 6 into an intermediate language (PIL). PIL-Run was built on a set of Perl 5 modules that reads PIL and executes it. PIL-Run evolved into a Pugs backend, invoked with “pugs -Bperl5.” The object subsystem, mostly developed by Stevan Little, eventually evolved into Perl 5 Moose.

LREP — Perl 6 bootstrap. LREP was a really small Perl 6 program capable of reading itself and emitting executable code. LREP evolved into Pugs::Compiler and later MiniPerl6.

Pugs::Compiler. These Perl 5 modules provide Perl6-regex support for Pugs. The regex runtime executes in Perl 5; the regex grammar is written in Perl 6 and runs in Perl 5 using a variant of Perl 6 bootstrap.

v6.pm (v6-pugs, later v6-alpha, now v6.pm) is a complete rewrite of Pugs, using a mix of Perl 6 (from Pugs::Compiler and friends) and Perl 5. It reuses some runtime code from PIL-run, and “half of CPAN.”

See also:

v6 is a CPAN distribution.

At some point, this distribution was very powerful, but it accumulated an enormous amount of code and became unmaintainable. Nowadays, v6.pm is distributed with Perlito6 instead.

v6-pugs is the original distribution name in CPAN. It implements a pluggable system, which you can use to run an arbitrary Perl 6 compiler.

The original v6::pugs module was a “reimplementation” of Pugs written with Perl 5 instead of Haskell. It was later renamed v6::alpha. Nowadays, v6.pm is distributed with v6::perlito instead of v6::alpha.

MiniPerl6 was a full compiler rewrite. The main reason was to clean up and add more structure. Audrey and I met during YAPC::SA::2006 and defined what the compiler would look like. I had written some ideas on an envelope, so we continued writing on the envelope until we could get online.

We defined syntax tree nodes more or less based on the planned PIL2 (from Pugs). It supported multiple backends from the start. Performance was more important than features. Source code would be 100% Perl 6, with minimal emitter modules that generate code for the supported VMs.

KindaPerl6 and SMOP. KindaPerl6 was meant to be a “full Perl 6” built on top of MiniPerl6.

See also:

I was working with Daniel Ruoso at the time KindaPerl6 was being developed, and he got interested in the object system. He developed a C backend, SMOP. SMOP was later combined with the STD.pm grammar, creating the “Mildew” compiler.

KindaPerl6 was distributed in CPAN for some time. The distribution included an SBCL Lisp backend. KindaPerl6 was eventually merged with MiniPerl6 to become Perlito6.

Perlito6 is basically a much refactored MiniPerl6. It is no longer limited by the bootstrap limitations. Instead, it is limited by reduced developer time.

Perlito5 is a port of Perlito6 back to Perl 5. It is a Perl 5 implemented in Perl 5. One of the things it does is to translate Perl 5 to Perl 6.

Perlito6 and Perlito5 are in the same repository.

Note that there are many more Perl 6 compilers — these are just the ones that led to Perlito.

Before Perlito, you worked on Pugs. What did you do there?

I worked on the Perl 5 backend for Pugs. Pugs has this feature that you can plug it into different execution environments. So we developed a Perl 5 environment to run Perl 6. Pugs compiles Perl 6 into an intermediate language (PIL), and a Perl 5 module reads PIL and executes it.

Later on, I was working in bootstrapping Perl 6, that is, writing the Perl 6 compiler in Perl 6 (instead of Haskell or Perl 5).

What is the purpose of the Pugs::Compiler::* modules?

Pugs::Compiler::Rule is on CPAN to provide Perl6-regex support for Pugs. The regex runtime executes in Perl 5.

Pugs::Compiler::Perl6 is the module that provides the Perl 6 compiler for “v6-alpha.” This module is no longer on CPAN. It is still available on BackPan.

Launching the project

Why did you start a separate project instead of continuing contributing to Pugs?

MiniPerl6 evolved into a separate source tree inside Pugs. At some point, there were lots of commits to the MiniPerl6 directory in Pugs, and this looked like noise to people monitoring the “original” Pugs. MiniPerl6 commits were huge because of how the bootstrapping process works — there is a lot of generated code. At the same time, people involved in MiniPerl6 were not much interested in the Haskell code changes.

So the code was moved to a separate repository; it was a reorganization. Also, after the bootstrap was complete, there was no reason to keep it “mini”, so it was renamed.

Perlito is designed to compile the source code into the abstract syntax tree (AST) and then generate code for one of the backends, like JavaScript, Perl 5, or even Haskell. Why did you need that variety?

Pugs had support for these backends, and it was a good thing. So we kept those and added even more.

As a compiler, does Perlito’s architecture differ from what it was in Pugs?

No, the compiler architecture is approximately the same. But the implementation looks very different just because they are written in different languages.

Having Haskell in Pugs might prevent other developers from joining. How easy is it for newcomers to dig into Perlito? For example, the Perlito5 X64 assembler internals could seem as tough as Haskell.

Having a clean code base is a high priority, and I believe Perlito is pretty readable. Certainly, understanding the bootstrapping process requires some effort.

The Perl 5 and JavaScript backends should be easy for newcomers. Perlito itself is written in Perl. Perlito6 is written in Perl 6, and Perlito5 is written in Perl 5.

Note that the X64 backend is a work in progress. I’m trying out the code architecture derived from the V8 JavaScript JIT, which is C++ code. It is likely still going to change a lot.

Yes, we had the idea that Haskell was a barrier for new developers. I’m not sure how much that was true — I’d say that changing the language just moved the barrier to the next step.

How many contributors are there in the project now, and how active are they? How do they communicate?

Nowadays, there are no active contributors; people send patches now and then using GitHub.

Sometimes, when I saw a comment on a forum about Perlito, I would contact the person, and we would work together for a while on a particular problem.

Most of the time, it is just people saying it would be cool to have a particular feature. This kind of contribution is more important than it looks — it directs where the project is going in the long run.

Language Design

Perlito implements only the subset of the Perl 6 language specification. Namely, the “useful subset” (subconjunto útil). How do you determine what is useful and what is not?

This was mostly defined by the bootstrapping process. We really had to implement the things that we needed for bootstrapping.

We bootstrapped the compilers. In terms of functionality, the focus has then moved into getting Perlito5 to be able to run CPAN modules.

On perl6.org, there’s a feature matrix comparing how different compilers cover the whole Perl 6 design. Why is there no Perlito data there?

I think Pugs and v6-alpha used to be there and maybe Mildew and KindaPerl6 at some point.

There is a bigger list of Perl 6 compilers at perl6.org/compilers.

Can you please explain the sources behind the list of differences between Perl 6 and its subset, aimed to be implemented in MiniPerl6?

  • no list contexts;
  • no inheritance;
  • no laziness;
  • closures, but no coroutines nor continuations;
  • no multis.

We really had to implement the things that we needed for bootstrapping and leave as much as possible out. That was just to make things possible; we had to simplify at that point.

On the other hand, there are features that are hard to implement efficiently if the VM doesn’t support it. List context can be implemented in JavaScript but with a small performance hit. Variable aliasing is also missing, and working around has a huge performance impact.

In the Perl 6 world, there is another example of a subset language, NQP (Not Quite Perl), used as intermediate mean in Rakudo. How different are these two subsets?

The intermediate language idea (the MiniPerl6 and KindaPerl6 pair of languages) seem to have inspired the creation of NQP.

In Perlito, we pretty much abandoned the idea of an intermediate language in favor of incremental bootstrapping. So Perlito6 tends to be more like Rakudo and less like NQP in the long run.

Perlito5 is pretty much just “perl;” there was never a mini-language involved in the process.

Perl 5

There are two Perlitos: Perlito5 and Perlito6, the Perl 5 and Perl 6 compilers, respectively. For Perl 6, multiple compilers should exist by design. But why did you need yet another Perl 5 compiler?

A few years ago, I attended a talk by Jesse Vincent.


Near slide 278, he was talking about reimplementing Perl 5.

I found it a nice challenge, and he convinced me that this would be a good thing for Perl 5. I converted most of the Perlito grammar and runtime over the night, so I had something to show him the next day of the conference.

Does the AST produced by Perlito5 differ from the one of Perlito6 (at least where there are no big differences between Perl 5 and Perl 6 features)?

The AST nodes are pretty much the same. There are some annotations (metadata) to the AST nodes that are different. Note that the semantics are very different; you can’t just evaluate a Perl 5 AST in a Perl 6 VM; things have different meanings.

For example, DESTROY works differently in Perl 6; it doesn’t do timely destruction in Perl 6.

Backends and Internals

The Haskell backend was not completed, right?

We found it was not necessary to have a complete Haskell backend. We had a way to call Perl 5 from Haskell, so we used that.

There was a backend for Parrot. Why did you need that? What did it generate?

The Parrot backend generated PIR, which is a little higher level than PASM (Parrot assembler). Parrot support was very important at that time because we believed it would be the ultimate VM for Perl 6.

What other backends did you consider? For example, were there any attempts to create ones for JVM (Java virtual machine) or MoarVM?

The backends that are working are JavaScript, Perl 5, and Python 2.7. These are bootstrapped backends — you can run the compiler in it.

We used to have bootstrapped backends in SBCL Lisp and in Go. The Go backend stopped working due to the changes in the language. That code was written when Go was very young, and the language has changed a lot since then. I’m not sure about the state of the Lisp backend.

The other backends are experimental and in different states of completion: Clojure, Java, Parrot, Ruby, XS, X64, and Rakudo. The Rakudo compiler was meant to rewrite some constructs that were missing in Rakudo; I think this is no longer a problem. Rakudo should be able to execute Perlito6 directly, but I haven’t tried recently.

How did you work with file input and output? Would they occur in a Perl 6 programme converted to JavaScript?

The Perl6 compiler will generate the function calls. You have to provide the functions, the runtime library.

How did you implement the eval keyword? (For that, you need a Perl 6 compiler running on a backend!)

Yes, the Perl 6 compiler must be present at run-time.

An eval call generates a lot of work. At compile time, we have to save the compiler state such as which variables are declared at that point. At runtime, we retrieve that compiler state and send the Perl 6 string to the Perl6-to-JavaScript compiler. The compiler returns the JavaScript source code, and we run a JavaScript eval to execute it.

Online

On perlito.org, there are a couple of awesome services: JavaScript compilers of Perl 5 and Perl 6 to JavaScript so that you can execute Perl 6 in a browser. With small test programs, it works pretty quickly. Do you have any statistics comparing its speed with modern versions of Rakudo, for example?

No, I don’t have those statistics. It is hard to compare because of the differences in the feature sets.

There’s also an online cross compiler producing the Perl 6 code from the Perl 5 source. It might be used for learning purposes, but even for trivial Perl 5 programs, it produces the code in Perl 6 with classes. Why do you need this Java-style behaviour?

use v6;
class main {
    say('Hello, World!')
}

This is a not-yet-implemented simplification step.

The Perl5-to-Perl6 compiler works by successively applying transformation rules over the syntax tree, and then it feeds the resulting tree into a pretty-printer. The class main structure is a representation of how this tree is stored in memory. What we need is to add another transformation step to simplify that structure.

How well is Perlito compatible with different JavaScript engines, like V8, or the one in PhantomJS, or with Node.js?

Node.js is my developing platform, so no problem there.

There are some smartphone browsers that can’t execute the perlito.org pages, but I haven’t investigated much. Chrome works fine in those same phones.

I haven’t tried PhantomJS.

The online Perl 5 and Perl 6 demos are great things for interactive tutorials. You discussed this idea somewhen [with Gabor Szabo], are there any plans for that?

This is doable up to some point. We currently can’t load arbitrary CPAN modules in the browser. If a module is preloaded, there is no problem: Data:: Dumper and “strict” work, for example. DBI doesn’t work. There is also the problem of emitting the exact same error messages as “Perl.” I’m working on this. On the other hand, there is the whole thing about developing the tutorial content. I’m not working on it, but I think it is important.

Future

What is the roadmap for the Perlito compiler set?

There are things that I find important, but I’m not committing to a roadmap. Instead, I maintain the TODO lists that I update from time to time:

I actually spend most of the time experimenting with unplanned things anyway.

Do you have any plans to extend the list of useful features that Perlito6 can work with?

Yes, there are a number of “low-hanging fruit” features that can be implemented in Perlito6. This includes updating the grammar to the latest specifications.

On the experimental side, interoperation between Perl 5 and Perl 6 would be an interesting thing to try.

Did you think about potential applications of Perlito, for example, embedded applications?

Yes, this would be nice to have, and I have discussed this with people involved in embedded applications. Martin Berends was interested in a possible Perl in Arduino.

Some discussion is here — there may be a market for that:

One thought on “Interview with Flávio Glock”

  1. Pingback: Old Perl- and Raku-related interviews – Andrew Shitov’s Blog

Leave a Reply

Your email address will not be published. Required fields are marked *

Retype the CAPTCHA code from the image
Change the CAPTCHA codeSpeak the CAPTCHA code