This is a chapter from
Creating a compiler with Raku
The compiler becomes bigger and bigger, and we are going to add more syntax to the Lingua language. It is a good time to make a pause in the compiler development and ensure some stability. In this chapter, we will create a test suite, which will allow us to find problems in the implementation and prevent new bugs when extending the language.
The test suite of Raku itself is a set of files written in Raku. They use the Test module similar to how we did in in Chapter 3. This is great but to run those tests, you need a compiler that can parse it. For our purpose, this approach is an overhead, so let us come up with our own solution.
Our goal is to test all features of Lingua, so we have to create test examples for variable declaration—both scalars, arrays, and hashes,—assignments, for different expressions with different operators, sting interpolation, etc. In the following chapters, we will add conditionals and function, so we will create tests for them too.
What were we doing in the previous chapters to see if the compiler works correctly? We were looking at the AST, at the content of the variable storage, and at the output. All three parts tell us about the health of the compiler. Testing all of them is a difficult task. Let us restrict ourselves and only check what the test programs output. The structure of the syntax tree is not that important for the end user, so we can check it manually time to time if needed. Variable storage is more important, but we always can print the variables and thus convert the second task into checking the output.
So, our final goal is to make the way to describe our desired output, and create a program that runs the examples and compares the output with the correct values.
Test runner
Let us start with a simple program that creates a variable and assigns a numeric value to it. We have at least two ways to do that:
my a;
a = 10;
my b = 20;
This program does not print anything, so let us add a couple of say calls.
my a;
a = 10;
say a;
my b = 20;
say b;
Save the test in a file, for example, variable-declaration.t, in a dedicated directory t and run it. It generates the result mixed with some test output (e.g., a full AST and a dump of the %!var hash) that we added to the compiler earlier. As we are still going to work on the compiler, let us keep the helper output but filter it out to /dev/null. The good thing is that Rakudo’s dd routine prints its output to STDERR, not to STDOUT.
$ ./lingua t/variable-declaration.t 2>/dev/null
OK
10
20
This looks clean, and we even can omit the OK line by using the note function instead of a conventional say:
if $ast {
note 'OK';
$evaluator.eval($ast.made);
}
Now, all the helper output goes to STDERR, and the program payload is printed via STDOUT.
$ ./lingua t/variable-declaration.t 2>/dev/null
10
20
This output is what we have to compare with the correct result. The simplest way is to save it in another file and just compare the files. The other option is to type the answers in the test program itself in the comments. To allow the standard comments, which are the user comments explaining the program, let us introduce our special type of comments with a double hash sign. It is still ignored by the compiler, but is visible for the test framework.
my a;
a = 10;
say a; ## 10
my b = 20;
say b; ## 20
The task is trivial now. We need to create a script that runs all the files in the t directory, extracts the comments from them, and compares the expected data with the real output. To simplify the task even further, let us assume that the test files are small programs, so if something goes wrong, you can easily spot the line where the error happened.
Let us create another executable program, run-tests. It scans the t directory and runs the tests one by one. We are using the Test module, which was introduced in Chapter 3.
#!/usr/bin/env perl6
use Test;
test-file($_) for dir('t').sort;
done-testing;
Running a single test is quite straightforward. The following function does the job using the built-in run function that places the output from STDOUT and STDERR to the out and err attributes of a return object.
sub test-file($file) {
my $path = $file.path;
my $proc = run('./lingua', $path, :out, :err);
is(expected($path), $proc.out.slurp, $path);
}
Escape sequences in the output add colours so that you can immediately see red FAILs or green OKs.
There are two possibilities to fail a test: either the file was not parsed, or it generated the output which differs from expectations. The expected output string is extracted from the ## comments:
sub expected($path) {
my $expected;
for $path.IO.lines -> $line {
$line ~~ /'##'\s*(\N*)/;
next unless $0;
$expected ~= "$0\n";
}
return $expected;
}
Run the program and you should see the following output:
ok 1 - t/array-creation.t ok 2 - t/expressions.t ok 3 - t/expressions2.t ok 4 - t/hash-creation.t ok 5 - t/hashes.t ok 6 - t/numbers.t ok 7 - t/print.t ok 8 - t/string-escaping.t ok 9 - t/string-index.t ok 10 - t/string-interpolation.t ok 11 - t/string.t ok 12 - t/variable-as-index.t ok 13 - t/variable-declaration.t 1..13
Now we can add more tests to cover more features of Lingua.
Tests
Having the test runner ready, it is time to fill the t directory with many files featuring different aspects of the language. One test file, variable-declaration.t is already there.
Numbers
Creating numbers in different formats:
my int = 42; say int; ## 42 my float = 3.14; say float; ## 3.14 my sci = 3E14; say sci; ## 300000000000000 my negative = -1.2; say negative; ## -1.2 my zero = 0; say zero; ## 0 my half = .5; say half; ## 0.5 my minus_n = -.3; say minus_n; ## -0.3
Notice that the floating-point numbers without the integer part are printed with a zero before the decimal point, and thus you should expect 0.5 instead of .5.
Strings
For strings, let us first test simple initialisation and assignment:
my s1; s1 = "Hello, World!"; say s1; ## Hello, World! my s2 = "Another string"; say s2; ## Another string my s3 = ""; # Empty string say s3; ##
In a separate file, let us test string interpolation:
my i = 10; my f = -1.2; my c = 1E-2; my s = "word"; my str = "i=$i, f=$f, c=$c, s=$s"; say str; ## i=10, f=-1.2, c=0.01, s=word
We also allow the three characters (\, ", and $) to be escaped:
say "\\"; ## \ say "\\\\"; ## \\ say "\$"; ## $ say "\""; ## "
Expressions
We’ve seen a lot of different tests for arithmetic expressions in Chapter 3. Gather them together and put in t/expression.t (or better in two or three different files) together with the answers. We should test all five operators in different order so that we check the precedence and parentheses.
Arrays and hashes
Aggregate data types are very important for the language, so let us test them too. First, a few trivial test cases for creating and printing arrays:
my a[]; a = 3, 4, 5; say a; ## 3, 4, 5 my b[] = 7, 8, 9; say b; ## 7, 8, 9
And hashes:
my h{};
say h; ##
my g{} = "a": "b", "c": "d";
say g; ## a: b, c: d
Then try using variables as array indices or hash keys:
my a[] = 2, 4, 6, 8, 10;
my i = 3;
say a[i]; ## 8
my b{} = "a": 1, "b": 2;
my j = "b";
say b{j}; ## 2
This test works fine.
Spotting the errors
The above tests all passed but let us try simple string indexing:
my abc = "abcdef"; say abc[0]; ## a say abc[3]; ## d say abc[5]; ## f
This time, the test fails with the following report:
not ok 17 - t/string-index.t # Failed test 't/string-index.t' # at ./run-tests line 14 # expected: 'a # d # f # ' # got: 'abcdef'
Instead of printing a single character, we have the whole string printed (and only once). Let us modify the program so that it prints the errors if the test failed.
is(expected($path), $proc.out.slurp, $path)
or say $proc.err.slurp;
In the error message, you’ll see some explanation that comes from Raku:
# Index out of range. Is: 3, should be in 0..0
We screwed the index and either did not save it properly in the AST node or did not use it later. Here is the fragment of the tree corresponding to the first two printing instructions:
AST::FunctionCall.new(
function-name => "say",
value => AST::ArrayItem.new(
variable-name => "abc",
index => AST::NumberValue.new(
value => 0
)
)
),
AST::FunctionCall.new(
function-name => "say",
value => AST::ArrayItem.new(
variable-name => "abc",
index => AST::NumberValue.new(
value => 3
)
)
),
The first assumption is not confirmed, as we see that the index is stored in the corresponding attribute of the AST node. Let us switch to the evaluator and see how the value is used there. Our candidate is a multi-method taking an array item:
multi method call-function('say', AST::ArrayItem $item) {
say %!var{$item.variable-name}[$item.index.value];
}
The parser understood the code as an array index, and the call-function method treats a string as an array. That works when the index is 0, because in Raku, you can always ‘index’ a scalar value:
$ raku To exit type 'exit' or '^D' > my $x = 10; 10 > say $x[0]; 10
With a non-zero index, the above-seen error pops up:
> say $x[3];
Index out of range. Is: 3, should be in 0..0
in block <unit> at <unknown file> line 1
Indeed, having an expression abc[3], how you can tell if the variable abc is an array or a string? We got rid of any sigils to make the language cleaner but now it would be nice to get them back. With a sigil, it is much easier to distinguish between $abc and @abc. You can go even further and use some kind of Hungarian notation to let the compiler know the type of the variable. Say, $s_abc stores strings only, and $i_abc only integers. Although it helps the compiler, it makes the language heavier for the end user.
Let us ask a computer to check the type of the variable at runtime. We did that already for the alternatives of multi-functions. Here is the updated version that only handle strings:
multi method call-function('say', AST::ArrayItem $item
where %!var{$item.variable-name} ~~ Str) {
say %!var{$item.variable-name}.substr($item.index.value, 1);
}
Run the test suite, and you’ll confirm that the test is passing. Unfortunately, another test, t/variable-as-index.t, becomes broken.
This example demonstrates how helpful testing can be. Without the test, you could change something in the compiler without noticing that something else starts working incorrectly. Fortunately, we discovered it pretty quickly, and the solution is just to return the previously modified multi-method that covers the missing code:
multi method call-function('say', AST::ArrayItem $item
where %!var{$item.variable-name} ~~ Array) {
say %!var{$item.variable-name}[$item.index.value];
}
The test suite is now reporting all fine.
Feature coverage
It is important to create the tests for every feature of the language and for as many potential use cases as possible. For example, we tested simple scalar assignments such as i = 10, but did not try to have an array items on the right side of the equation:
my a[] = 3, 5, 7; my x = a[1]; say x;
Surprisingly, that does not work and produces the error message:
No such method 'value' for invocant of type 'AST::ArrayItem'. Did you mean 'values'? in method eval-node at /Users/ash/lingua/LinguaEvaluator.rakumod (LinguaEvaluator) line 17 in method eval at /Users/ash/lingua/LinguaEvaluator.rakumod (LinguaEvaluator) line 9 in block <unit> at ./lingua line 21
The problem is at the line with the assignment. The AST tree is built correctly:
. . .
AST::ScalarDeclaration.new(
variable-name => "x",
value => AST::ArrayItem.new(
variable-name => "a",
index => AST::NumberValue.new(
value => 1
)
)
),
. . .
Let us check the evaluator (it is clearly referred by the error message, by the way).
multi method eval-node(AST::ScalarDeclaration $node) {
%!var{$node.variable-name} =
$node.value ~~ AST::Null ?? 0 !! $node.value.value;
}
Everything is fine here, but we do not have the value for the AST::ArrayItem node, and thus we have to implement it:
class AST::ArrayItem is ASTNode {
has Str $.variable-name;
has ASTNode $.index;
has $.evaluator;
method value() {
return $.evaluator.var{$!variable-name}[$!index.value];
}
}
You also might notice that we don’t have such a method in the AST::HashItem class. Add it too:
class AST::HashItem is ASTNode {
has Str $.variable-name;
has ASTNode $.key;
has $.evaluator;
method value() {
return $.evaluator.var{$!variable-name}{$!key.value};
}
}
Don’t forget to pass an evaluator when building the AST nodes in the actions class (there is more than one place in the LinguaActions.rakumod file):
$/.make(AST::ArrayItem.new(
. . .
evaluator => $!evaluator,
));
. . .
$/.make(AST::HashItem.new(
. . .
evaluator => $!evaluator,
));
We should add tests for both arrays and hashes:
my g{} = "a": "b", "c": "d";
say g{"a"}; ## b
my x = g{"a"};
say x; ## b
And that (almost) concludes the chapter. All the tests are passing again!
It is never enough
Before making the next step, let us fix one tiny element that we introduced earlier. Consider the following assignments with expressions.
my x = 10;
my y = 2 * x;
my z = x * 2;
say "$x, $y, $z";
There should be no difference between the values assigned to y and z, but the parse method refuses to accept the second expression, x * 2. This happens at the place where a variable catches control in the grammar:
rule value {
| <variable-name> <index>?
| <expression>
| <string>
}
In fact, as we have the AST now, it is fine to let the variable be a part of an expression, and we can safely remove that alternative:
rule value {
| <expression>
| <string>
}
This small change restores the parser, and both 2 * x and x * 2, as well as x * x can be parsed now.
In such cases, it is important to not only fix the grammar, actions or evaluator, but also to create a regression test to avoid the same bug in the future:
my x = 10; my y = 2 * x; my z = x * 2; my q = x * x; my d = 2 * 2; say "$x, $y, $z"; ## 10, 20, 20 say "$q, $d"; ## 100, 4
Next: Chapter 10. Control Flow
One thought on “Chapter 10. Test suite”