PHPT: Writing tests for PHP

This year the PHP Community promoted the PHPTestFest in various places around the globe and it was a success. I participated in the event promoted by the PHPSP UG which ended up as the team with the most committed tests, after participating in the event I kept on going and have since obtained a SVN account and karma to be a official test commiter to PHP. So if you did not get the chance to learn how to write tests in your local TestFest I will now go over the steps taken so you can be ready for next year.

PHPT is a extremely simple test framework designed to test PHP. Its atomic nature and independent test execution make it perfect for the kind of tests needed, so tests can be really focused and test specific function and/or bugs.

What do I need to know?

The beauty about PHPT is that you need to know very little other than writing PHP code. A little knowledge into the inner workings of PHP will of course help you in finding areas of code that need testing, and how best to test them, but just knowing PHP is enough to start.

I have divided this post into 5 parts so we can get started:

  1. Preparing the Environment
  2. Choosing what to test
  3. Writing a test
  4. Executing a test
  5. Submitting a test to PHP

1. Preparing the Environment

Your test environment needs the correct version of PHP and the ability to run tests, I’m going to focus on *nix/MacOS systems hereA  as I have not had experience with tests on windows. The first step is to compile PHP from source, during the testfest we focused on 5.3 but you should head over to the PHP QA Page (qa.php.net) to see what version is being targeted or take you pick from the PHP SVN. Anyhow you need to download the source code for your system.

The basic compilation process means you need to run ./configure with your correct settings and run make. After running make you are already ready to run tests, using the make test command which will run all the current tests (9000+) and prompt you to send a report to the QA Team, you should do it. But before you begin writing your own tests i recommend that you also run make install so you can have a proper php executable of the version you are testing, it might be useful. Later on we will see how to use make test to target only the tests you wish to.

2. Choosing what to test

Before you begin writing a test you must decide on what you are writing a test for. You can usually pick from two points to write tests for, you will be either writing a test for a specific bug or to cover areas of PHP without “code coverage”. To figure out the bugs that have/need tests you should get on the QA mailing list or check out the correct path (Ex: For GD go to ext/gd/tests) for a filename with the bug-id.

To write tests to cover untouched areas of the source code your best bet is to head over to http://gcov.php.net where you should pick you branch/version and look for th areas with less code coverage. In this year’s testfest the PHPSP group focused on the GD extension, so we where basically looking at the coverage for the gd.c file, which i will use for examples in this article. To decide what to test took a little bit of bit-scrubbing, so this is where i commented that knowing the inner working of PHP would help, but after a few examples and tests you can easily pickup what needs to be done.

Let’s start by analyzing these lines of code from gd.c:

PHPT-Gcov

What we are looking for here are the red lines, which means that these lines were not executed. The main objective is to get all lines to either blue (executed) or white (non-executable code) as you can see above. First thing you should notice here is that this is the code for the imageclorallocatealpha function as displayed online 1848 by the PHP_FUNCTION construct.

Now you should look at the red lines, we can easily see that line 1856 is never executed, and judging by the IF construct before it we can state that this line is never executed because the zend_parse_parameters function (nevermind what it does for now) never returns the FAILURE state. As it happens, the zend_parse_parameters is responsible for parsing and validating the parameters passed to the function, so we can guess that this means that the current tests never pass an invalid parameter to the function, thus never testing if it actually returns a proper error message. This seems like  good place to begin learning about tests.

3. Writing a test

Writing the actual test is usually really simple, and it actually should be since tests need to be as atomic as possible. You should limit the scope of your test trying to test only the function at hand and the situation defined to be tested, this will also help you name you test as we will see next.

3.1 Naming test files

Naming test files is simple if you defined you scope correctly:

  • Bug test: bug<bugid>.phpt
  • Basic function usage: <function>_basic.phpt
  • Function error behaviours: <function>_error.phpt
  • Variations of function usage: <function>_variation.phpt
  • Extensions: <extension>_<#>.phpt

3.2 Structure of a .phpt file

The simple structure of PHPT tests follows this basic pattern:

--TEST--
strtr() function - basic test for strstr()
--FILE--
<?php
/* Do not change this test it is a README.TESTING example. */
$trans = array("hello"=>"hi", "hi"=>"hello", "a"=>"A", "world"=>"planet");
var_dump(strtr("# hi all, I said hello world! #", $trans));
?>
--EXPECT--
string(32) "# hello All, I said hi planet! #"

As we can see the –TEST– section holds a one-liner title for the test and should make it pretty clear as to what is being tested. The –FILE– section is the actual test code, this will be internally converted into a .php file and executed. And finally the –EXPECT– section will be compared to the .php output to see if they are the same, var_dump is recommended for the tests.

These are just the most standard sections the full documentation can be checked on the QA site under phpt format, and its recommended reading.

You will come across situation where he above blocks are not enough, but generally you will be checking if X function is returning Y value when passed Z parameters so the above is usually all you need. But on block is very important and i should mention it , its the –CREDIT– block. Putting your name here is very important so you can be reached in case of problems and get your name on the credits for writing tests in PHP. We used this format in our testfest:

--CREDIT--
Rafael Dohms <myemail@gmail.com>
#PHPSPTestFest 2009-04-DD

Getting back to the code above, let’s try to write a test to cover the untested line we selected above. Let’s look closely at the parse function and figure out what we need to test:

zend_parse_parameters(ZEND_NUM_ARGS() TSRMLS_CC, "rllll", &IM, &red, &green, &blue, &alpha)

The important part to look at is the “rllll” as it defines the type needed for each of the parameters. The first parameter for example requires a Resource as stated by the “r”, whereas the rest of them need Long values, if we look at the PHP manual for this function we will get:

int imagecolorallocatealpha ( resource $image , int $red , int $green , int $blue , int $alpha )

It confirms what we saw in the code, but it might also show wrong information, so this would be another point where you can help PHP updating and fixing the documentation. But this is not the case here, so let’s get back to the test. Let’s write a test to make sure that the function sends an error message if we pass something that is not a Resource as its first parameter.

Looking inside the ext/gd/tests folder we do not find any imagecolorallocatealpha files, so according to the naming conventions above we should use the name imagecolorallocatealpha_error1.phpt, for our test file, since we are checking error conditions and it is the first error test for this function. (in sequence we can user error2, error3… for the other parameters)

--TEST--
Testing imagecolorallocatealpha(): Wrong types for parameter 1
--CREDITS--
Rafael Dohms <rdohms [at] gmail [dot] com>

First step is to get the test title right and tag it with our credit so we can be found in case of future changes or problems. Make sure your title describes the objective really well without having to write an essay.

--SKIPIF--
<?php
if (!extension_loaded("gd")) die("skip GD not present");
?>

The –SKIPIF– section was not mentioned before but its a key section in this case. This is very important because we are testing a function of the GD extension, so this means that depending on the local environment it might actually not be available, if GD is not enabled for example. In this case we do not want to execute these tests if the server has no GD otherwise they would all fail, which might give us some false negatives. This section can also be used for other skip criteria, like 32 bit Vs. 64 bit tests, or any condition that you feel might affect your test.

The section itself is very simple, just do a check and in case the conditions for skipping are met, call a die command and the test will be marked as SKIP not FAIL in the final execution.

--FILE--
<?php
$resource = tmpfile();

imagecolorallocatealpha($resource, 255, 255, 255, 50);
imagecolorallocatealpha('string', 255, 255, 255, 50);
imagecolorallocatealpha(array(), 255, 255, 255, 50);
imagecolorallocatealpha(null, 255, 255, 255, 50);
?>

This is the body of the actual test and as we mentioned its really simple and atomic. What we did basically is to call the function various times with different types of parameters. In the first call we are actually testing line 1859, because tmpfile returns a valid Resource, however not an Image Resource and line 1859 should catch that. The rest of the parameters receive valid values, in this case integers. Testing various types is interesting because you might just find a specific type that causes non-expected behaviour. Since we are hoping for error messages, we do not even need the return value of the function.

--EXPECTF--
Warning: imagecolorallocatealpha(): supplied resource is not a valid Image resource in %s on line %d

Warning: imagecolorallocatealpha() expects parameter 1 to be resource, %s given in %s on line %d

Warning: imagecolorallocatealpha() expects parameter 1 to be resource, array given in %s on line %d

Warning: imagecolorallocatealpha() expects parameter 1 to be resource, null given in %s on line %d

Finally we now have a –EXPECTF– block which is the same as a –EXPECT– block but uses a printf-like structure to skip out on dynamic parts of the test. As we can see above pieces like file name or line are replaced by %s or %d meaning it will fit any value, this is important because different environments might return different file paths and such, so you would get a lot of FAILs due to this.

You can also use blocks with regular expressions, but this format is much more simple and easy to read. As we can see we are expecting an error message for each of the calls we made.

Tip: To get these error messages, execute the phpt file using a php executable, like: php imagecolorallocatealpha_error1.phpt. The file will be executed and everything outside <? tags will be outputted, but the actual output for the –FILE– block will be outputted as well, just copy-paste and replace dynamic bits.

Tip 2: the error message usually states the given type, as in: string given; If you are expecting the string type in the error message, replace it with %s to make your test compatible with PHP 5.3- and PHP6, since in this case it would print “unicode string” instead of “string”

4. Executing a test

Your test is ready for the drill, now we need the drillmaster to pick it up and make it jump through the hoops. We already saw that make test executes the PHP tests, but with 9000+ tests this takes a while, way too much time to figure out you left out a “,” and need to do it all again. So we should get it to execute only our test, or related tests we need. To do this we can use the TESTS parameter which will define which tests should be executed:

make test TESTS=ext/gd/tests/imagecolorallocatealpha_*.phpt

You can either pass it the full path to your tst or use a wildcard (*) like above to execute all tests that match the pattern, this is useful if we decided to write tests for all parameters and want to run them all at once.

This is the resulting output:

Test Results

The test passes as we can see by the PASS text present before the test title (–TEST– block). Its also noteworthy to state that if this test had failed, the test framework would create a few extra files for you to debug the problem, these files have the same name as the original .phpt but different extensions:

  • .out – the output for the php code
  • .php – source code generated for the test (–FILE–)
  • .diff – a diff between –EXPECT– and the actual output
  • .exp – the isolated text in the –EXPECT– section
  • .log – a log of all information related (output, expected…)

These files are extremely useful in debugging and are auto-removed once you get the test to PASS.

5. Submitting you test to PHP

There are a few ways to get your test in the PHP SVN. The easiest one is to participate in a TestFest event, all tests written at these events are submitted to a separate repository and then migrated to the official repository. The second way is to send your test file to the QA mailing list (use pastebin) and get someone to review it. The final way is a mix of the previous ones, participate in a TestFest event.. get the taste for tests write a whole bunch and at the end, apply for a SVN account, if you play your cards right and keep on writing tests you will easily get Karma and will be a official test commiter in PHP.  Hey it happened to me, true story, so it just depends on you and your will to write tests.

Conclusion

Now you have all the tools you need and proper orientation, what are you waiting for? Go help out the PHP Community. But don’t stop there, get the taste for tests and bring them to your application as well, but then i recommend your look into PHPUnit and TDD, which i hope to write more about soon.