PHPUnit has hinted at parallelism since 2007, but, in the meantime, our tests continue to run slowly. Time is money, right? ParaTest is a tool that sits on top of PHPUnit and allows you to run tests in parallel without the use of extensions. This is an ideal candidate for functional (i.e Selenium) tests and other long-running processes.

ParaTest at your Service

ParaTest is a robust command line tool for running PHPUnit tests in parallel. Inspired by the fine folks at Sauce Labs, it was originally developed to be a more complete solution for improving the speed of functional tests.

Since its inception - and thanks to some brilliant contributors (including Giorgio Sironi, the maintainer of the PHPUnit Selenium extension) - ParaTest has become a valuable tool for speeding up functional tests, as well as integration tests involving databases, web services, and file systems.

ParaTest also has the honor of being bundled with Sauce Labs' testing framework Sausage, and has been used in nearly 7000 projects, at the time of this writing.

Installing ParaTest

Currently, the only official way to install ParaTest is through Composer. For those of you who are new to Composer, we have a great article on the subject. To fetch the latest development version, include the following within your composer.json file:

"require": {
    "brianium/paratest": "dev-master"
}

Alternatively, for the latest stable version:

"require": {
    "brianium/paratest": "0.4.4"
}

Next, run composer install from the command line. The ParaTest binary will be created in the vendor/bin directory.

The ParaTest Command Line Interface

ParaTest includes a command line interface that should be familiar to most PHPUnit users - with some added bonuses for parallel testing.

Your First Parallel Test

Using ParaTest is just as simple as PHPUnit. To quickly demonstrate this in action, create a directory, paratest-sample, with the following structure:

Let's install ParaTest as mentioned above. Assuming that you have a Bash shell and a globally installed Composer binary, you can accomplish this in one line from the paratest-sample directory:

echo '{"require": { "brianium/paratest": "0.4.4" }}' > composer.json && composer install

For each of the files in the directory, create a test case class with the same name, like so:

class SlowOneTest extends PHPUnit_Framework_TestCase
{
    public function test_long_running_condition()
    {
        sleep(5);
        $this->assertTrue(true);
    }
}

Take note of the use of sleep(5) to simulate a test that will take five seconds to execute. So we should have five test cases that each take five seconds to run. Using vanilla PHPUnit, these tests will run serially and take twenty-five seconds, total. ParaTest will run these tests concurrently in five separate processes and should only take five seconds, not twenty-five!

Now that we have an understanding of what ParaTest is, let's dig a little deeper into the problems associated with running PHPUnit tests in parallel.

The Problem at Hand

Testing can be a slow process, especially when we start talking about hitting a database or automating a browser. In order to test more quickly and efficiently, we need to be able to run our tests concurrently (at the same time), instead of serially (one after the other).

The general method for accomplishing this is not a new idea: run different test groups in multiple PHPUnit processes. This can easily be accomplished using the native PHP function proc_open. The following would be an example of this in action:

/**
 * $runningTests - currently open processes
 * $loadedTests - an array of test paths
 * $maxProcs - the total number of processes we want running
 */
while(sizeof($runningTests) || sizeof($loadedTests)) {
    while(sizeof($loadedTests) && sizeof($runningTests) < $maxProcs)
        $runningTests[] = proc_open("phpunit " . array_shift($loadedTests), $descriptorspec, $pipes);
    //log results and remove any processes that have finished ....
}

Because PHP lacks native threads, this is a typical method for acheiving some level of concurrency. The particular challenges of testing tools that use this method can be boiled down to three core problems:

• How do we load tests?
• How do we aggregate and report results from the different PHPUnit processes?
• How can we provide consistency with the original tool (i.e PHPUnit)?

Let's look at a few techniques that have been employed in the past, and then review ParaTest and how it differs from the rest of the crowd.

Those Who Came Before

As noted earlier, the idea of running PHPUnit in multiple processes is not a new one. The typical procedure employed is something along the following lines:

• Grep for test methods or load a directory of files containing test suites.
• Open a process for each test method or suite.
• Parse output from STDOUT pipe.

Let's take a look at a tool that employs this method.

Hello, Paraunit

Paraunit was the original parallel runner bundled with Sauce Labs' Sausage tool, and it served as the starting point for ParaTest. Let's look at how it tackles the three main problems mentioned above.

Test Loading

Paraunit was designed to ease functional testing. It executes each test method rather than an entire test suite in a PHPUnit process of its very own. Given the path to a collection of tests, Paraunit searches for individual test methods, via pattern matching against file contents.

preg_match_all("/function (test[^\(]+)\(/", $fileContents, $matches);

Loaded test methods can then be run like so:

proc_open("phpunit --filter=$testName $testFile", $descriptorspec, $pipes);

In a test where each method is setting up and tearing down a browser, this can make things quite a bit faster, if each of those methods is run in a separate process. However, there are a couple of problems with this method.

While methods that begin with the word, "test," is a strong convention among PHPUnit users, annotations are another option. The loading method used by Paraunit would skip this perfectly valid test:

/**
 * @test
 */
public function twoTodosCheckedShowsCorrectClearButtonText()
{
    $this->todos->addTodos(array('one', 'two'));
    $this->todos->getToggleAll()->click();
    $this->assertEquals('Clear 2 completed items', $this->todos->getClearButton()->text());
}

In addition to not supporting test annotations, inheritance is also limited. We might argue the merits of doing something like this, but let's consider the following setup:

abstract class TodoTest extends PHPUnit_Extensions_Selenium2TestCase
{
    protected $browser = null;

    public function setUp()
    {
       //configure browser
    }

    public function testTypingIntoFieldAndHittingEnterAddsTodo()
    {
        //selenium magic
    }
}

/**
 * ChromeTodoTest.php
 * No test methods to read!
 */
class ChromeTodoTest extends TodoTest
{
    protected $browser = 'chrome';
}

/**
 * FirefoxTodoTest.php
 * No test methods to read!
 */
class FirefoxTodoTest extends TodoTest
{
    protected $browser = 'firefox';
}

The inherited methods aren't in the file, so they will never be loaded.

Displaying Results

Paraunit aggregates the results of each process by parsing the output generated by each process. This method allows Paraunit to capture the full gamut of short codes and feedback presented by PHPUnit.

The downside to aggregating results this way is that it is pretty unwieldy and easy to break. There are a lot of different outcomes to account for, and a lot of regular expressions at work to display meaningful results in this way.

Consistency with PHPUnit

Due to the file grepping, Paraunit is fairly limited in what PHPUnit features it can support. It's an excellent tool for running a simple structure of functional tests, but, in addition to some of the difficulties mentioned already, it lacks support for some useful PHPUnit features. Some such examples include test suites, specifying configuration and bootstrap files, logging results, and running specific test groups.

Many of the existing tools follow this pattern. Grep a directory of test files and either run the entire file in a new process or each method - never both.

ParaTest At Bat

The goal of ParaTest is to support parallel testing for a variety of scenarios. Originally created to fill the gaps in Paraunit, it has become a robust command line tool for running both test suites and test methods in parallel. This makes ParaTest an ideal candidate for long running tests of different shapes and sizes.

How ParaTest Handles Parallel Testing

ParaTest deviates from the established norm in order to support more of PHPUnit, and acts as a truly viable candidate for parallel testing.

Test Loading

ParaTest loads tests in a similar manner to PHPUnit. It loads all tests in a specified directory that end with the *Test.php suffix, or will load tests based on the standard PHPUnit XML configuration file. Loading is accomplished, via reflection, so it is easy to support @test methods, inheritance, test suites and individual test methods. Reflection makes adding support for other annotations a snap.

Because reflection allows ParaTest to grab classes and methods, it can run both test suites and test methods in parallel, making it a more versatile tool.

ParaTest does impose some constraints, but well-founded ones in the PHP community. Tests do need to follow the PSR-0 standard, and the default file suffix of *Test.php is not configurable, as it is in PHPUnit. There is a current branch in progress for supporting the same suffix configuration allowed in PHPUnit.

Displaying Results

ParaTest also deviates from the path of parsing STDOUT pipes. Instead of parsing output streams, ParaTest logs the results of each PHPUnit process in the JUnit format and aggregates results from these logs. It's much easier to read test results from an established format than an output stream.

<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
  <testsuite name="AnotherUnitTestInSubLevelTest" file="/home/brian/Projects/parallel-phpunit/test/fixtures/tests/level1/AnotherUnitTestInSubLevelTest.php" tests="3" assertions="3" failures="0" errors="0" time="0.005295">
    <testcase name="testTruth" class="AnotherUnitTestInSubLevelTest" file="/home/brian/Projects/parallel-phpunit/test/fixtures/tests/level1/AnotherUnitTestInSubLevelTest.php" line="7" assertions="1" time="0.001739"/>
    <testcase name="testFalsehood" class="AnotherUnitTestInSubLevelTest" file="/home/brian/Projects/parallel-phpunit/test/fixtures/tests/level1/AnotherUnitTestInSubLevelTest.php" line="15" assertions="1" time="0.000477"/>
    <testcase name="testArrayLength" class="AnotherUnitTestInSubLevelTest" file="/home/brian/Projects/parallel-phpunit/test/fixtures/tests/level1/AnotherUnitTestInSubLevelTest.php" line="23" assertions="1" time="0.003079"/>
  </testsuite>
</testsuites>

Parsing JUnit logs does have some minor drawbacks. Skipped and ignored tests are not reported in the immediate feedback, but they will be reflected in the totaled values displayed after a test run.

Consistency with PHPUnit

Reflection allows ParaTest to support more PHPUnit conventions. The ParaTest console supports more PHPUnit features out of the box than any other similar tool, such as the ability to run groups, supply configuration and bootstrap files, and log results in the JUnit format.

ParaTest Examples

ParaTest can be used to gain speed in several testing scenarios.

Functional Testing With Selenium

ParaTest excels at functional testing. It supports a -f switch in its console to enable functional mode. Functional mode instructs ParaTest to run each test method in a separate process, instead of the default, which is to run each test suite in a separate process.

It's often the case that each functional test method does a lot of work, such as opening a browser, navigating around the page, and then closing the browser.

The example project, paratest-selenium, demonstrates testing a Backbone.js todo application with Selenium and ParaTest. Each test method opens a browser and tests a specific feature:

public function setUp()
{
    $this->setBrowserUrl('http://backbonejs.org/examples/todos/');
    $this->todos = new Todos($this->prepareSession());
}

public function testTypingIntoFieldAndHittingEnterAddsTodo()
{
    $this->todos->addTodo("parallelize phpunit tests\n");
    $this->assertEquals(1, sizeof($this->todos->getItems()));
}

public function testClickingTodoCheckboxMarksTodoDone()
{
    $this->todos->addTodo("make sure you can complete todos");
    $items = $this->todos->getItems();
    $item = array_shift($items);
    $this->todos->getItemCheckbox($item)->click();
    $this->assertEquals('done', $item->attribute('class'));
}

//....more tests

This test case could take a hot second if it were to run serially, via vanilla PHPUnit. Why not run multiple methods at once?

Handling Race Conditions

As with any parallel testing, we have to be mindful of scenarios that will present race conditions - such as multiple processes trying to access a database. The dev-master branch of ParaTest sports a really handy test token feature, written by collaborator Dimitris Baltas (dbaltas on Github), that makes integration testing databases much easier.

Dimitris has included a helpful example that demonstrates this feature on Github. In Dimitris' own words:

TEST_TOKEN attempts to deal with the common resources issue in a very simple way: clone the resources to ensure that no concurrent processes will access the same resource.

A TEST_TOKEN environment variable is provided for tests to consume ,and is recycled when the process has finished. It can be used to conditionally alter your tests, like so:

public function setUp()
{
    parent::setUp();
    $this->_filename = sprintf('out%s.txt', getenv('TEST_TOKEN'));
}

ParaTest and Sauce Labs

Sauce Labs is the Excalibur of functional testing. Sauce Labs provides a service that allows you to easily test your applications in a variety of browsers and platforms. If you haven't checked them out before, I strongly encourage you to do so.

Testing with Sauce could be a tutorial in itself, but those wizards have already done a great job of providing tutorials for using PHP and ParaTest to write functional tests using their service.

The Future of ParaTest

ParaTest is a great tool for filling in some of the gaps of PHPUnit, but, ultimately, it's just a plug in the dam. A much better scenario would be native support in PHPUnit!

In the meantime, ParaTest will continue increasing support for more of PHPUnit's native behavior. It will continue to offer features that are helpful to parallel testing - particularly in the functional and integration realms.

ParaTest has many great things in the works to beef up the transparency between PHPUnit and itself, primarily in what configuration options are supported.

The latest stable version of ParaTest (v0.4.4) comfortably supports Mac, Linux, and Windows, but there are some valuable pull requests and features in dev-master that definitely cater to the Mac and Linux crowds. So that will be an interesting conversation moving forward.

Additional Reading and Resources

There are a handful of articles and resources around the web that feature ParaTest. Give them a read, if you're interested:

• ParaTest on Github
• Parallel PHPUnit by ParaTest contributor and PHPUnit Selenium extension maintainer Giorgio Sironi
• Contributing to Paratest. An excellent article on Giorgio's experimental WrapperRunner for ParaTest
• Giorgio's WrapperRunner Source Code
• tripsta/paratest-sample. An example of the TEST_TOKEN feature by it's creator Dimitris Baltas
• brianium/paratest-selenium. An example of using ParaTest to write functional tests