Test Anything Protocol

The Test Anything Protocol (TAP) is a general purpose format for transmitting the result of test programs to a thing which interprets and takes action on those results. Though it is language agnostic, it is primarily used by Perl modules.

TAP's general format is:

   1..N
   ok 1 Description # Directive
   # Diagnostic
   ....
   ok 47 Description
   ok 48 Description
   more tests....

For example, a test file's output might look like:

   1..4
   ok 1 - Input file opened
   not ok 2 - First line of the input valid
   ok 3 - Read the rest of the file
   not ok 4 - Summarized correctly # TODO Not written yet

TAP separates the test program from the thing which interprets the results using simple text pipelining. This allows one to run several tests isolated in different processes together in one test harness. It allows one to write tests in different languages and run them together with unified results. It also allows the runner of the tests, rather than the author of the tests, to decide how to display the results. The results can be transformed by the runner into other formats (for example, XML) or even cause actions to happen (a smoke test server can send email on failure).

TAP is designed to be human readable. The simple "ok", "not ok" output and descriptions do not require a parser or even extensive knowledge of TAP for the basic intention of each line to be understood by the user.

TAP is designed to be portable. Its plain text format means it will work with any language or operating system.

TAP is designed to be streamable. The results can be interpreted a line at a time without having to wait for the test to complete.

Specification

Despite its being almost 20 years old and in very wide use, it is only now being formally specified. This is to allow for adoption by other languages, the writing of additional parsers, and extension of TAP. The current spec is the TAP module on CPAN. This spec is under development and incomplete. Where the spec is lacking, the behavior of Test::Harness with the existing Perl core tests and, to a lesser extent, CPAN module tests, should be used as a guide.

A formal grammar is in development.

History

TAP came into being with the first version of Perl (1987) although it wasn't called anything then. For most of its existence the only TAP parser was Perl's core test harness (t/TEST) written by Larry Wall. When Perl 5 introduced modules, Test::Harness was written by Tim Bunce and Andreas König allowing Perl module authors to take advantage of TAP. Testing has become very popular in Perl in the last few years and Perl authors working with other languages wishing to take their tools with them have begun to write TAP parsers and TAP-based testing libraries in other languages.

TAP Parsers

These are libraries which parse TAP and display the results.

• Test::Harness is the oldest and most complete TAP parser. It is limited in how it displays TAP. Though it most often runs tests written in Perl, it can launch any process which generates TAP. Most of the TAP spec is taken from the behavior of Test::Harness.
  • The original Test::Harness has now been deprecated, the new Test::Harness provides a minimal compatibility layer with previous behavior, but any new development shouldn't use this module, rather the TAP::Harness module.
• The t/TEST parser contained in the Perl source code.
• TAP::Parser is a new and more flexible parser being written by Curtis "Ovid" Poe, Andy Armstrong and other people. It was formerly called TAPx::Parser. It will shortly become Test::Harness 3.
• Test::Run is a fork of Test::Harness being written by Shlomi Fish.
• test-harness.php A TAP parser for PHP.

TAP Producers

These are libraries for writing tests which output TAP.

• Test::More is the most popular testing module for Perl.
• PHPUnit is the xUnit implementation for PHP.
• test-more.php is a testing module for PHP based on Test::More.
• libtap is a TAP producer written in C.
• Test.Simple is a port of the Perl Test::Simple and Test::More modules to JavaScript by David Wheeler.
• PyTAP A beginning TAP implementation for Python.
• MyTAP MySQL unit test library used for writing TAP producers in C or C++
• Bacon A Ruby library that supports a spec-based syntax and that can produce TAP output
• PLUTO PL/SQL Unit Testing for Oracle
• pgTAP PostgreSQL stored procedures that emit TAP

Other Resources

• testanything.org is a site dedicated to the discussion, development and promotion of TAP.