ANGLE Test Harness

The ANGLE test harness is a harness around GoogleTest that provides functionality similar to the Chromium test harness. It features:

• splitting a test set into shards
• catching and reporting crashes and timeouts
• outputting to the Chromium JSON test results format
• multi-process execution

Command-Line Arguments

The ANGLE test harness accepts all standard GoogleTest arguments. The harness also accepts the following additional command-line arguments:

• --batch-size limits the number of tests to run in each batch
• --batch-timeout limits the amount of time spent in each batch
• --bot-mode enables multi-process execution and test batching
• --debug-test-groups dumps the test config categories when using bot-mode
• --filter-file allows passing a larger gtest_filter via a file
• --histogram-json-file outputs a formatted JSON file for perf dashboards
• --max-processes limits the number of simuntaneous processes
• --results-directory specifies a directory to write test results to
• --results-file specifies a location for the JSON test result output
• --shard-count and --shard-index control the test sharding
• --test-timeout limits the amount of time spent in each test
• --flaky-retries allows for tests to fail a fixed number of times and still pass
• --disable-crash-handler forces off OS-level crash handling
• --isolated-outdir specifies a test artifacts directory
• --max-failures specifies a count of failures after which the harness early exits.

--isolated-script-test-output and --isolated-script-perf-test-output mirror --results-file and --histogram-json-file respectively.

As well as the custom command-line arguments we support a few standard GoogleTest arguments:

• gtest_filter works as it normally does with GoogleTest
• gtest_also_run_disabled_tests works as it normally does as well

Other GoogleTest arguments are not supported although they may work.

Implementation Notes

• The test harness only requires angle_common and angle_util.
• It does not depend on any Chromium browser code. This allows us to compile on other non-Clang platforms.
• It uses rapidjson to read and write JSON files.
• Test execution is not currently deterministic in multi-process mode.

Normal Mode vs Bot Mode

The test runner has two main modes of operation: normal and bot mode.

During normal mode:

• Tests are executed single-process and single-thread.
• The test runner executes the GoogleTest Run function.
• We use a TestEventListener to record test results for our output JSON file.
• A watchdog thread will force a fast exit if no test results get recorded after a timeout.
• Crashes are handled via ANGLE's test crash handling code.

During bot mode:

• Tests are run in multiple processes depending on the system processor count.
• A server process records the child processes' stdout and stderr.
• The server terminates a child process if there's no progress after a timeout.
• The server sorts work into batches according to the back-end configuration.
• This prevents driver errors from using multiple back-ends in the same process.
• Batches are striped to help split up slow groups of tests.
• The server passes test batches to child processes via a gtest_filter file.
• Bot mode does not work on Android or Fuchsia.

See the source code for more details: TestSuite.h and TestSuite.cpp.

Potential Areas of Improvement

• Deterministic test execution.
• Using sockets to communicate with test children. Similar to dEQP's test harness.
• Closer integration with ANGLE's test expectations and system config libraries.
• Supporting a GoogleTest-free integration.