🦋 108. Basic usage of NativeCall in Perl 6

NativeCall is both a module and a technology in Perl 6 that allows you to call C functions from your Perl 6 code. Today, let’s meet the most basic usage.

Take the rand() function from the C standard library:

#include <stdio.h>
#include <stdlib.h>

int main() {    
    int r = rand();
    printf("%i\n", r);

    return 0;
}

If you are used to Perl, it may be a surprise that the program actually prints the same number every time you call it. (That’s a kind of surprise of forgotten knowledge.) Compile and run:

$ gcc rand.c

$ ./a.out 
16807

$ ./a.out 
16807

Now let’s call C’s rand() from Perl 6. Refer to the documentation of NativeCall to see the options it offers.

Your Perl 6 program may look like this:

use NativeCall;

sub c_rand() returns int32 is native('c') is symbol('rand') {*}

say c_rand();

Although the rand() function was chosen as one from the standard library that do not need parameters, the choice added a few small complications to the Perl 6 code. But it is good that we can demonstrate them all together.

Similarly to how you declare external functions in C, you need to tell Perl 6 that there is some function that it should load from an external library. The name of the library is passed in the native trait. Its argument, c, will be converted to libc, and the corresponding shared library will be searched for in the standard places on your computer.

The symbol trait tells us the original function name. It is a bit weird to me, but Rakudo cannot handle the following code where you use the same name as in C:

use NativeCall;

sub rand() returns int32 is native('c') {*}

say rand();

This is how you would declare an external C function if you didn’t want to rename it in your program. Unfortunately, Perl 6 also has rand, and we got an error:

===SORRY!=== Error while compiling rand.pl6
 Unsupported use of rand(); in Perl 6 please use rand
 at rand.pl6:5
 ------> say rand⏏();

Finally, returns int32 tells that the function returns a native 32-bit integer.

Another surprise pops up when you run our Perl 6 program. It returns a random value, and that value is different each time you call it:

$ perl6 rand.pl6 
1790432239

$ perl6 rand.pl6 
1431059869

Is it really working? It is random, but why it does not copy the behaviour of the reference C program?

Does Rakudo call srand when it starts up? A shallow investigation gives the following log comment:

nqp/MoarVM/docs/ChangeLog:+ Do not call srand() if not using rand()

Let’s go back to our test case. To make sure it works, let us create our own version of the function, which will be returning the same value again and again:

myrand.c:

int myrand() {
    return 42;
}

myrand.h:

int myrand();

Test it in pure C first:

#include "myrand.h"
#include <stdio.h>

int main() {
    int r = myrand();
    printf("%i\n", r);

    return 0;
}
$ gcc myrand-main.c myrand.c 

$ ./a.out 
42

$ ./a.out 
42

It works. Now use it from Perl 6, and the program is almost the same as before:

use NativeCall;

sub myrand() returns int32 is native('libmyrand.so') {*}

say myrand();

The main differences are the name of the function (there is no clash with Perl 6 built-in functions, and thus no need to introduce an alias) and the library name, which is a file name this time.

(This is how you create a shared library:)

$ gcc -shared -olibmyrand.so myrand.c 

If you run the updated Perl 6 program, you will only see 42 in the output.

$ perl6 myrand.pl6 
42

$ perl6 myrand.pl6 
42

This confirms that the returns int32 clause works correctly, and the value returned from the C library is understood by Perl 6.

🦋 74. Typed hashes in Perl 6

In Perl 6, you can restrict the content of a variable container by specifying its type, for example:

my Int $i;

There is only one value in a scalar variable. You can extend the concept to arrays and let its element to keep only integers, as it is done in the next example:

> my Int @i;
[]

> @i.push(42);
[42]

> @i.push('Hello');
Type check failed in assignment to @i;
expected Int but got Str ("Hello")
  in block <unit> at <unknown file> line 1

Hashes keeps pairs, so you can specify the type of both keys and values. The syntax is not deductible from the above examples.

First, let us announce the type of the value:

my Str %s;

Now, it is possible to have strings as values:

> %s<Hello> = 'World'
World

> %s<42> = 'Fourty-two'
Fourty-two

But it’s not possible to save integers:

> %s<x> = 100
Type check failed in assignment to %s;
expected Str but got Int (100)
  in block <unit> at <unknown file> line 1

(By the way, notice that in the case of %s<42> the key is a string.)

To specify the type of the second dimension, namely, of the hash keys, give the type in curly braces:

my %r{Rat};

This variable is also referred to as object hash.

Having this, Perl expects you to have Rat keys for this variable:

> %r<22/7> = pi
3.14159265358979

> %r
{22/7 => 3.14159265358979}

Attempts to use integers or strings, for example, fail:

> %r<Hello> = 1
Type check failed in binding to parameter 'key';
expected Rat but got Str ("Hello")
  in block <unit> at <unknown file> line 1

> %r{23} = 32
Type check failed in binding to parameter 'key';
expected Rat but got Int (23)
  in block <unit> at <unknown file> line 1

Finally, you can specify the types of both keys and values:

my Str %m{Int};

This variable can be used for translating month number to month names but not vice versa:

> %m{3} = 'March'
March

> %m<March> = 3
Type check failed in binding to parameter 'key';
expected Int but got Str ("March")
  in block <unit> at <unknown file> line 1

 

🦋 69. Setting timeouts in Perl 6

In Perl 5, I used to set timeouts using signals (or, at least, that was an easy and predictable way). In Perl 6, you can use promises. Let us see how to do that.

To imitate a long-running task, create an infinite loop that prints its state now and then. Here it is:

for 1 .. * {
    .say if $_ %% 100_000;
}

As soon as the loop gets control, it will never quit. Our task is to stop the program in a couple of seconds, so the timer should be set before the loop:

Promise.in(2).then({
    exit;
});

for 1 .. * {
    .say if $_ %% 100_000;
}

Here, the Promise.in method creates a promise that is automatically kept after the given number of seconds. On top of that promise, using then, we add another promise, whose code will be run after the timeout. The only statement in the body here is exit that stops the main program.

Run the program to see how it works:

$ time perl6 timeout.pl
100000
200000
300000
. . .
3700000
3800000
3900000

real 0m2.196s
user 0m2.120s
sys 0m0.068s

The program counts up to about four millions on my computer and quits in two seconds. That is exactly the behaviour we needed.

For comparison, here is the program in Perl 5:

use v5.10;

alarm 2;
$SIG{ALRM} = sub {
    exit;
};

for (my $c = 1; ; $c++) {
    say $c unless $c % 1_000_000;
}

(It manages to count up to 40 million, but that’s another story.)

🦋 66. Atomic operations in Perl 6

N. B. The examples below require a fresh Rakudo compiler, at least of the version 2017.09.

Discussing parallel computing earlier or later leads to solving race conditions. Let us look at a simple counter that is incremented by two parallel threads:

my $c = 0;

await do for 1..10 {
    start {
        $c++ for 1 .. 1_000_000
    }
}

say $c;

If you run the program a few times, you will immediately see that the results are very different:

$ perl6 atomic-1.pl 
3141187
$ perl6 atomic-1.pl 
3211980
$ perl6 atomic-1.pl 
3174944
$ perl6 atomic-1.pl 
3271573

Of course, the idea was to increase the counter by 1 million in all of the ten threads, but about ⅓ of the steps were lost. It is quite easy to understand why that happens: the parallel threads read the variable and write to it ignoring the presence of other threads and not thinking that the value can be changed in-between. Thus, some of the threads work with an outdated value of the counter.

Perl 6 offers a solution: atomic operations. The syntax of the language is equipped with the Atom Symbol (U+0x269B) ⚛ character (no idea of why it is displayed in that purple colour). Instead of $c++, you should type $c⚛++.

my atomicint $c = 0;

await do for 1..10 {
    start {
        $c⚛++ for 1 .. 1_000_000
    }
}

say $c;

And before thinking of the necessity to use a Unicode character, let us look at the result of the updated program:

$ perl6 atomic-2.pl 
10000000

This is exactly the result we wanted!

Notice also, that the variable is declared as a variable of the atomicint type. That is a synonym for int, which is a native integer (unlike Int, which is a data type represented by a Perl 6 class).

It is not possible to ask a regular value to be atomic. That attempt will be rejected by the compiler:

$ perl6 -e'my $c; $c⚛++'
Expected a modifiable native int argument for '$target'
  in block  at -e line 1

A few other operators can be atomic, for example, prefix and postfix increments and decrements ++ and --, or += and -=. There are also atomic versions of the assignment operator = and the one for reading: ⚛ (sic!).

If you need atomic operations in your code, you are not forced to use the ⚛ character. There exists a bunch of alternative functions that you can use instead of the operators:

my atomicint $c = 1;

my $x = ⚛$c;  $x = atomic-fetch($c);
$c ⚛= $x;     atomic-assign($c, $x);
$c⚛++;        atomic-fetch-inc($c);
$c⚛--;        atomic-fetch-dec($c);
++⚛$c;        atomic-inc-fetch($c);
--⚛$c;        atomic-dec-fetch($c);
$c ⚛+= $x;    atomic-fetch-add($c,$x);

say $x; # 1
say $c; # 3

🦋64. What does gist do in Perl 6?

When you print an object, say, as say $x, Perl 6 calls the gist method. This method is defined for all built-in types: for some of them, it calls the Str method, for some the perl method, for some types it makes the string representation somehow differently.

Let us see how you can use the method to create your own variant:

class X {
    has $.value;

    method gist {
        '[' ~ $!value ~ ']'
    }
}

my $x = X.new(value => 42);

say $x; # [42]
$x.say; # [42]

When you call say, the program prints a number in square brackets: [42].

Please notice that the interpolation inside double-quoted strings is using Str, not gist. You can see it here:

say $x.Str; # X<140586830040512>
say "$x";   # X<140586830040512>

If you need a custom interpolation, redefine the Str method:

class X {
    has $.value;

    method gist {
        '[' ~ $!value ~ ']'
    }
    method Str {
        '"' ~ $!value ~ '"'
    }
}

my $x = X.new(value => 42);

say $x;     # [42]
$x.say;     # [42]

say $x.Str; # "42"
say "$x";   # "42"

Now, we got the desired behaviour.

🦋63. More on the proto keyword in Perl 6

Before digging into the details of the EVAL routine, we have to reveal some more information about protos and multiple dispatch. Examine the following program:

proto sub f($x) {
    say "proto f($x)";
}

multi sub f($x) {
    say "f($x)"
}

multi sub f(Int $x) {
    say "f(Int $x)"
}

multi sub f(Str $x) {
    say "f(Str $x)"
}

f(2);
f('2');
f(3);
f('3');

Here, there are three multi-candidates of the function plus a function declared with the proto keyword. Earlier, we only saw such proto-functions with empty body, such as:

proto sub f($x) {*}

But this is not a necessity. The function can carry a regular load, as we see in the example:

proto sub f($x) {
    say "proto f($x)";
}

Run the program:

proto f(2)
proto f(2)
proto f(3)
proto f(3)

All the calls were caught by the proto-candidate. Now, update it and return the {*} block for some dedicated values;

proto sub f($x) {
    if $x.Str eq '3' {
        return {*}
    }
    say "proto f($x)";
}

The if check triggers its block for the last two function calls:

f(3);
f('3');

In these cases, the proto-function returns {*}, which makes Perl 6 trying other candidates. As we have enough candidates for both integer and string arguments, the compiler can easily choose one of them:

proto f(2)
proto f(2)
f(Int 3)
f(Str 3)

🦋62. The EVAL routine in Perl 6, part 1

The EVAL routine in Perl 6 compiles and executes the code that it gets as an argument.  Today, we will see some potential use cases that you may try in your practice. Tomorrow, we will dig into Rakudo sources to see how it works and why it breaks sometimes.

1

Let us start with evaluating a simple program:

EVAL('say 123');

This program prints 123, there’s no surprise here.

2

There are, though, more complicated cases. What, do you think, does the following program print?

EVAL('say {456}');

I guess it prints not what you expected:

-> ;; $_? is raw { #`(Block|140570649867712) ... }

It parses the content between the curly braces as a pointy block.

3

What if you try double quotes?

EVAL("say {789}");

Now it even refuses to compile:

===SORRY!=== Error while compiling eval.pl
EVAL is a very dangerous function!!! (use the MONKEY-SEE-NO-EVAL pragma to override this error,
but only if you're VERY sure your data contains no injection attacks)
at eval.pl:6
------> EVAL("say {789}")⏏;

4

We can fix the code by adding a few magic words:

use MONKEY-SEE-NO-EVAL;

EVAL("say {789}");

This time, it prints 789.

5

The code is executed (we don’t know yet when exactly, that is the topic of tomorrow’s post), so you can make some calculations, for example:

use MONKEY-SEE-NO-EVAL;

EVAL("say {7 / 8 + 9}"); # 9.875

6

Finally, if you try passing a code block directly, you also cannot achieve the goal, even with a blind monkey:

use MONKEY-SEE-NO-EVAL;

EVAL {say 123};

The error happens at runtime:

Constraint type check failed in binding to parameter '$code';
expected anonymous constraint to be met but got 
-> ;; $_? is raw { #`...
  in block <unit> at eval.pl line 10

This message looks cryptic, but at least we see once again that we got an anonymous pointy block passed to the function.

7

And before we wrap up for today, an attempt to use Perl 5 syntax:

eval('say 42');

There is no such function in Perl 6, and we get a standard error message:

===SORRY!=== Error while compiling eval2.pl
Undeclared routine:
  eval used at line 5. Did you mean 'EVAL', 'val'?

It looks OK but it can be better.

Stay tuned, tomorrow we will try to understand how all these examples work in Rakudo.

🦋17. Parameterised roles in Perl 6

Today, a small excursus into the syntax. Did you know that roles in Perl 6 can have a parameter that makes them similar to generic templates in, say, C++? Here’s a small example:

role R {
    has $.value;
 
    method add($b) {
        $.value + $b.value
    }

    method div($b) {
        $.value / $b.value
    }
}

The R role defines an interface that has a value and two methods for arithmetical operations: add and div.

Now, create a class using the role, initialise two variables and use the methods to get the results:

class C does R {}

my C $x = C.new(value => 10);
my C $y = C.new(value => 3);

say $x.add($y); # 13
say $x.div($y); # 3.333333

Although the values here were integers, Perl did a good job and returned a rational number for the division. You can easily see it by calling the WHAT method:

say $x.add($y).WHAT; # (Int)
say $x.div($y).WHAT; # (Rat)

If you have two integers, the result of their division is always of the Rat type. The actual operator, which is triggered in this case, is the one from src/core/Rat.pm:

multi sub infix:</>(Int \a, Int \b) {
    DIVIDE_NUMBERS a, b, a, b
}

The DIVIDE_NUMBERS sub returns a Rat value.

Defining a role

How to modify the C class so that it performs integer division? One of the options is to use a parameterised role:

role R[::T] {
    has T $.value;
    
    method add($b) {
        T.new($.value + $b.value)
    }

    method div($b) {   
        T.new($.value / $b.value)
    }
}

The parameter in square brackets after the role name restricts both the type of the $.value attribute and the return type of the methods, which return a new object of the type T. Here, in the template of the role, T is just a name, which should later be specified when the role is used.

Using the role

So, let’s make it integer:

class N does R[Int] {}

Now the parts of the role that employ the T name replace it with Int, so the class is equivalent to the following definition:

class C {
    has Int $.value;
    
    method add($b) {
        Int.new($.value + $b.value)
    }

    method div($b) {   
        Int.new($.value / $b.value)
    }
}

The new class operates with integers, and the result of the division is an exact 3:

class N does R[Int] {}

my N $i = N.new(value => 10);
my N $j = N.new(value => 3);

say $i.add($j); # 13
say $i.div($j); # 3

It is also possible to force floating-point values by instructing the role accordingly:

class F does R[Num] {}

my F $x = F.new(value => 10e0);
my F $y = F.new(value => 3e0);

say $x.add($y); # 13
say $x.div($y); # 3.33333333333333

Notice that both values, including 13, are of the Num type now, not Int or Rat as it was before:

say $x.add($y).WHAT; # (Num)
say $x.div($y).WHAT; # (Num)

🦋11. Compiler stages and targets in Perl 6

Welcome to the new year! Today, let us switch for a while from the discussion about obsolete messages to something different.

Stages

If you followed the exercises in the previous posts, you might have noticed that some statistics was printed in the console when compiling Rakudo:

Stage start      :   0.000
Stage parse      :  44.914
Stage syntaxcheck:   0.000
Stage ast        :   0.000
Stage optimize   :   4.245
Stage mast       :   9.476
Stage mbc        :   0.200

You could have also noticed that the bigger the file you changed, the slower it is compiled, up to dozens of seconds when you modify Grammar.pm.

It is also possible to see the statistics for your own programs. The --stagestats command-line option does the job:

$ ./perl6 --stagestats -e'say 42'
Stage start      :   0.000
Stage parse      :   0.065
Stage syntaxcheck:   0.000
Stage ast        :   0.000
Stage optimize   :   0.001
Stage mast       :   0.003
Stage mbc        :   0.000
Stage moar       :   0.000
42

So, let’s look at these stages. Roughly, half of them is about Perl 6, and half is about MoarVM. In the case Rakudo is configured to work with the JVM backend, the output will differ in the second half.

The Perl 6 part is clearly visible in the src/main.nqp file:

# Create and configure compiler object.
my $comp := Perl6::Compiler.new();
$comp.language('perl6');
$comp.parsegrammar(Perl6::Grammar);
$comp.parseactions(Perl6::Actions);
$comp.addstage('syntaxcheck', :before);
$comp.addstage('optimize', :after);
hll-config($comp.config);
nqp::bindhllsym('perl6', '$COMPILER_CONFIG', $comp.config);

Look at the selected lines. If you have played with Perl 6 Grammars, you know that big grammars are usually split into two parts: the grammar itself and the actions. The Perl 6 compiler does exactly the same thing for the Perl 6 grammar. There are two files: src/Perl6/Grammar.nqp and src/Perl6/Actions.nqp.

When looking at src/main.nqp, it is not quite clear that there are eight stages. Add the following line to the file:

for ($comp.stages()) { nqp::say($_) }

Now, recompile Rakudo and run any program:

$ ./perl6 -e'say 42'
start
parse
syntaxcheck
ast
optimize
mast
mbc
moar
42

Here they are.

The names of the first three stages—start, parse, and syntaxcheck—are quite self-explanatory. The ast stage is the stage of building an abstract syntax tree, which is then optimized in the optimize stage.

At this point, your Perl 6 program has been transformed into the abstract syntax tree and is about to be passed to the backend, MoarVM virtual machine in our case. The stages names start with m. The mast stage is the stage of the MoarVM assembly (not abstract) syntax tree, mbc stands for MoarVM bytecode and moar is when the VM executes the code.

Targets

Now that we know the stages of the Perl 6 program workflow, let’s make use of them. The --target option lets the compiler to stop at the given stage and display the result of it. This option supports the following values: parse, syntaxcheck, ast, optimize, and mast. With those options, Rakudo prints the output as a tree, and you can see how the program changes at different stages.

Even for small programs, the output, especially with the abstract syntax tree or an assembly tree of the VM is quite verbose. Let’s look at the parse tree of the ‘Hello, World!’ program, for example:

$ ./perl6 --target=parse -e'say "Hello, World!"'
- statementlist: say "Hello, World!"
  - statement: 1 matches
    - EXPR: say "Hello, World!"
      - args:  "Hello, World!"
        - arglist: "Hello, World!"
          - EXPR: "Hello, World!"
            - value: "Hello, World!"
              - quote: "Hello, World!"
                - nibble: Hello, World!
      - longname: say
        - name: say
          - identifier: say
          - morename:  isa NQPArray
        - colonpair:  isa NQPArray

All the names here correspond to rules, tokens, or methods of the Grammar. You can find them in src/Perl6/Grammar.nqp. As an exercise, try predicting if the name is a method, or a rule, or a token. Say, a value should be a token, as it is supposed to be a compact string, while a statementlist is a rule.

🦋1. The proto keyword in Perl 6

Today, we are looking precisely at the proto keyword. It gives a hint for the compiler about your intention to create multi-subs.

Example 1

Consider an example of the function that either flips a string or negates an integer.

multi sub f(Int $x) {
    return -$x;
}

multi sub f(Str $x) {
    return $x.flip;
}

say f(42);      # -42
say f('Hello'); # olleH

What if we create another variant of the function that takes two arguments.

multi sub f($a, $b) {
    return $a + $b;
}

say f(1, 2); # 3

This code perfectly works, but it looks like its harmony is broken. Even if the name of the function says nothing about what it does, we intended to have a function that somehow returns a ‘reflected’ version of its argument. The function that adds up two numbers does not fit this idea.

So, it is time to clearly announce the intention with the help of the proto keyword.

proto sub f($x) {*}

Now, an attempt of calling the two-argument function won’t compile:

===SORRY!=== Error while compiling proto.pl
Calling f(Int, Int) will never work with proto signature ($x)
at proto.pl:15
------> say f(1,2)

The calls of the one-argument variants work perfectly. The proto-definition creates a pattern for the function f: its name is f, and it takes one scalar argument. Multi-functions specify the behaviour and narrow their expertise to either integers or strings.

Example 2

Another example involves a proto-definition with two typed arguments in the function signature.

proto sub g(Int $x, Int $y) {*}

In this example, the function returns a sum of the two integers. When one of the numbers is much bigger than the other, the smaller number is just ignored as being not significant enough:

multi sub g(Int $x, Int $y) {
   return $x + $y;
}

multi sub g(Int $x, Int $y where {$y > 1_000_000 * $x}) {
   return $y;
}

Call the function with integer arguments and see how Perl 6 picks the correct variant:

say g(1, 2);          # 3
say g(3, 10_000_000); # 10000000

Didn’t you forget that the prototype insists on two integers? Try it out passing floating-point numbers:

say g(pi, e);

We got a compile-time error:

===SORRY!=== Error while compiling proto-int.pl
Calling g(Num, Num) will never work with proto signature (Int $x, Int $y)
at proto-int.pl:13
------> say ⏏g(pi, e);

The prototype has caught the error in the function usage. What happens if there is no proto for the g sub? The function is still not called, but the error message is different. It happens at run-time this time:

Cannot resolve caller g(3.14159265358979e0, 2.71828182845905e0); none of these signatures match:
 (Int $x, Int $y)
 (Int $x, Int $y where { ... })
 in block <unit> at proto-int.pl line 13

We still have no acceptable signature for the floating-point numbers, but the compiler cannot see that until the program flow reaches the code.