Co-written by Jonathan “Duke” Leto, Parrot core developer and author of Tapir.
Introduction
I’ll be creating more complex PIR programs soon, but first I want to stop for a minute and look at testing in Parrot. Why? Code is a weird thing. You need to pin down its behavior as specifically as you can, or it’ll become unreadable before you realize what’s going on. Good tests help you describe how your program should behave. Tests aren’t a magic pill that will guarantee perfect programs, but they will help you check that your program behaves the way you claim it does.
There are many testing libraries in the programming world, but I will focus on Test::More for Parrot.
Using Test::More to Write Tests
Test::More is more or less an implementation of Perl’s Test::More. It provides a set of simple assertions such as ok
, is
, and isnt
, along with a few testing-specific commands like skip
and todo
. I’ll be looking at some of those simple assertions, but not spending so much time on the testing commands. This is a Babystep, after all.
Test::More is already included in the standard Parrot runtime, so we don’t need to do anything special to install it. Even better - there’s a test_more.pir include file that you can include to import all of the important Test::More subroutines automatically.
Let’s start writing tests.
plan
Every test needs a plan. The plan
subroutine in Test::More tells the world one simple thing: how many tests are in this file. Accuracy is important, because it’s no fun when you are told to expect ten tests but only five run. The other five might not have run for a number of reasons: the test script failed, Parrot failed in some mysterious way, or you just forgot to mention that you removed half of your tests.
We don’t plan to have any tests yet, so let’s be honest.
The .include
directive will insert the contents of test_more.pir
into the subroutine, which saves us a lot of namespace wrangling. The testing starts when a plan is declared.
Of course, this is not the most exciting test plan in the world to run.
What if we lie?
Running this is a little different.
Now Parrot is telling whoever cares that there will be ten tests in this file. It’s true that nothing exploded. For right now, you’re going to have to trust me when I say that honesty is the best policy. You’ll see later that some tools do care about how many tests you claim to run.
diag
All right. Sometimes we want to make a comment in our test for the world to see. We could just say
what we want to say, but Test::More provides the diag
subroutine to produce those comments in a manner that will make testers appy later.
What does this produce?
See the #
? That’s supposed to make our diagnostic comment stand out from the test results without confusing anyone. But the diagnostic makes me sad. Let’s write an actual test.
ok
ok
takes two arguments:
- The value you are testing
- A description of the test
The value being tested is obviously the most important part, but don’t underestimate the helpfulness of those descriptions. They are a form of documentation.
The test in ok
is one of simple truth as seen by Parrot. parrot-babysteps-03-simple-control-structures showed us that anything which looks like 0
or an empty string is considered false by Parrot, while everything else is considered true.
What happens when we introduce a test that we know will fail?
You updated your plan, right? Anyways, let’s see what this produces.
Oh hey, this is starting to get interesting! Now we can see clearly that the output from ok
is a line split into three parts:
- The result of the test: “
ok
” or “not ok
” - The test number
- Our description string
ok
has shown us what a test result line looks like. Let’s look at some of the other simple assertions.
nok
Sometimes you are more concerned if something is true which shouldn’t be. For example, let’s say we have a Web site building script. It builds temporary cache files to save time when building subpage links, but those cache files need to go away when it’s done. So we would test for existence of a cache file and fail if the file exists.
The assertion may be nok
, but the output is still ok
or not based on whether the assertion was true.
What does it look like if we deliberately confuse things?
Yes. That’s what I hoped to see. Let’s clean up after ourselves to avoid future confusion.
is
There are many times where we want to compare two values. Let’s continue with our Web site building tool. This tool sets the title of a page in metadata. We obviously want to be certain that it reads the metadata correctly. We would use the is
assertion for that kind of test.
Anybody know what we should see?
Let’s deliberately mess things up again so we know what failure of is
looks like.
A failed is
produces some useful information.
There’s the test result line, which shows ’not ok’, just like we expected. We also have a couple of diagnostic lines describing what we want and what we actually have.
ok
has its opposite assertion nok
, so there must be an opposite for is
, right? There sure is.
isnt
Occasionally we care less about what a value is than making sure it’s not something in particular. Maybe we have a user registration process that uses social security numbers to satisfy an obscure corporate tracking requirement, but can’t save them as-is because of privacy concerns. In this case we don’t care what the stored value is. We want to be certain that it’s not the social security number.
Really, nobody should be surprised by the output at this point.
What does a failed isnt
look like?
The output diagnostic is once again straightforward.
is_deeply
is
fails us when we need to compare PMCs. Well, it sort of works:
The output isn’t incredibly useful, though.
Thankfully, we have the is_deeply
assertion to tell use exactly how a test has failed.
Now we can see exactly which value in the PMC was different.
With is_deeply
under our belt, we now know enough assertions to get started putting them to use in real projects.
What About The Other Assertions and Commands?
We won’t be talking about them. I may eventually visit more as we get the hang of Parrot, but this is a good enough core to start with. Do you want to dig deeper? Go right ahead. The best resource for the moment is the documentation within Test::More itself.
TAP - The Test Anything Protocol
All of this output has looked remarkably consistent. There’s a reason for that. Test::More formats its result in a format known as TAP - the Test Anything Protocol. All of the output can be read by another program to provide you with a summary report. This other program is usually referred to as a test harness. The test harness runs your tests and then tells you how many of them failed, or if there were any surprises.
All I need is a test harness.
Conclusion
Hey, we can test now! We learned how to use the Test::More library, making simple assertions and reporting the results using the Test Anything Protocol. As long as we stay disciplined and run our tests regularly, we will learn immediately when we have an “inspired” moment that breaks existing code. Since I’m such a huge fan of Test-Driven Development, you can be assured of seeing many assertions in future Parrot Babysteps.
Backlinks
Added to vault 2024-01-15. Updated on 2024-01-26