[Summary view] [Print] [Text view]

   1  =head1 NAME
   2  
   3  Test::Harness::TAP - Documentation for the TAP format
   4  
   5  =head1 SYNOPSIS
   6  
   7  TAP, the Test Anything Protocol, is Perl's simple text-based interface
   8  between testing modules such as Test::More and the test harness
   9  Test::Harness.
  10  
  11  =head1 TODO
  12  
  13  Exit code of the process.
  14  
  15  =head1 THE TAP FORMAT
  16  
  17  TAP's general format is:
  18  
  19      1..N
  20      ok 1 Description # Directive
  21      # Diagnostic
  22      ....
  23      ok 47 Description
  24      ok 48 Description
  25      more tests....
  26  
  27  For example, a test file's output might look like:
  28  
  29      1..4
  30      ok 1 - Input file opened
  31      not ok 2 - First line of the input valid
  32      ok 3 - Read the rest of the file
  33      not ok 4 - Summarized correctly # TODO Not written yet
  34  
  35  =head1 HARNESS BEHAVIOR
  36  
  37  In this document, the "harness" is any program analyzing TAP output.
  38  Typically this will be Perl's I<prove> program, or the underlying
  39  C<Test::Harness::runtests> subroutine.
  40  
  41  A harness must only read TAP output from standard output and not
  42  from standard error.  Lines written to standard output matching
  43  C</^(not )?ok\b/> must be interpreted as test lines.  All other
  44  lines must not be considered test output.
  45  
  46  =head1 TESTS LINES AND THE PLAN
  47  
  48  =head2 The plan
  49  
  50  The plan tells how many tests will be run, or how many tests have
  51  run.  It's a check that the test file hasn't stopped prematurely.
  52  It must appear once, whether at the beginning or end of the output.
  53  
  54  The plan is usually the first line of TAP output and it specifies how
  55  many test points are to follow. For example,
  56  
  57      1..10
  58  
  59  means you plan on running 10 tests. This is a safeguard in case your test
  60  file dies silently in the middle of its run.  The plan is optional but if
  61  there is a plan before the test points it must be the first non-diagnostic
  62  line output by the test file.
  63  
  64  In certain instances a test file may not know how many test points
  65  it will ultimately be running. In this case the plan can be the last
  66  non-diagnostic line in the output.
  67  
  68  The plan cannot appear in the middle of the output, nor can it appear more
  69  than once.
  70  
  71  =head2 The test line
  72  
  73  The core of TAP is the test line.  A test file prints one test line test
  74  point executed. There must be at least one test line in TAP output. Each
  75  test line comprises the following elements:
  76  
  77  =over 4
  78  
  79  =item * C<ok> or C<not ok>
  80  
  81  This tells whether the test point passed or failed. It must be
  82  at the beginning of the line. C</^not ok/> indicates a failed test
  83  point. C</^ok/> is a successful test point. This is the only mandatory
  84  part of the line.
  85  
  86  Note that unlike the Directives below, C<ok> and C<not ok> are
  87  case-sensitive.
  88  
  89  =item * Test number
  90  
  91  TAP expects the C<ok> or C<not ok> to be followed by a test point
  92  number. If there is no number the harness must maintain
  93  its own counter until the script supplies test numbers again. So
  94  the following test output
  95  
  96      1..6
  97      not ok
  98      ok
  99      not ok
 100      ok
 101      ok
 102  
 103  has five tests.  The sixth is missing.  Test::Harness will generate
 104  
 105      FAILED tests 1, 3, 6
 106      Failed 3/6 tests, 50.00% okay
 107  
 108  =item * Description
 109  
 110  Any text after the test number but before a C<#> is the description of
 111  the test point.
 112  
 113      ok 42 this is the description of the test
 114  
 115  Descriptions should not begin with a digit so that they are not confused
 116  with the test point number.
 117  
 118  The harness may do whatever it wants with the description.
 119  
 120  =item * Directive
 121  
 122  The test point may include a directive, following a hash on the
 123  test line.  There are currently two directives allowed: C<TODO> and
 124  C<SKIP>.  These are discussed below.
 125  
 126  =back
 127  
 128  To summarize:
 129  
 130  =over 4
 131  
 132  =item * ok/not ok (required)
 133  
 134  =item * Test number (recommended)
 135  
 136  =item * Description (recommended)
 137  
 138  =item * Directive (only when necessary)
 139  
 140  =back
 141  
 142  =head1 DIRECTIVES
 143  
 144  Directives are special notes that follow a C<#> on the test line.
 145  Only two are currently defined: C<TODO> and C<SKIP>.  Note that
 146  these two keywords are not case-sensitive.
 147  
 148  =head2 TODO tests
 149  
 150  If the directive starts with C<# TODO>, the test is counted as a
 151  todo test, and the text after C<TODO> is the explanation.
 152  
 153      not ok 13 # TODO bend space and time
 154  
 155  Note that if the TODO has an explanation it must be separated from
 156  C<TODO> by a space.
 157  
 158  These tests represent a feature to be implemented or a bug to be fixed
 159  and act as something of an executable "things to do" list.  They are
 160  B<not> expected to succeed.  Should a todo test point begin succeeding,
 161  the harness should report it as a bonus.  This indicates that whatever
 162  you were supposed to do has been done and you should promote this to a
 163  normal test point.
 164  
 165  =head2 Skipping tests
 166  
 167  If the directive starts with C<# SKIP>, the test is counted as having
 168  been skipped.  If the whole test file succeeds, the count of skipped
 169  tests is included in the generated output.  The harness should report
 170  the text after C< # SKIP\S*\s+> as a reason for skipping.
 171  
 172      ok 23 # skip Insufficient flogiston pressure.
 173  
 174  Similarly, one can include an explanation in a plan line,
 175  emitted if the test file is skipped completely:
 176  
 177      1..0 # Skipped: WWW::Mechanize not installed
 178  
 179  =head1 OTHER LINES
 180  
 181  =head2 Bail out!
 182  
 183  As an emergency measure a test script can decide that further tests
 184  are useless (e.g. missing dependencies) and testing should stop
 185  immediately. In that case the test script prints the magic words
 186  
 187      Bail out!
 188  
 189  to standard output. Any message after these words must be displayed
 190  by the interpreter as the reason why testing must be stopped, as
 191  in
 192  
 193      Bail out! MySQL is not running.
 194  
 195  =head2 Diagnostics
 196  
 197  Additional information may be put into the testing output on separate
 198  lines.  Diagnostic lines should begin with a C<#>, which the harness must
 199  ignore, at least as far as analyzing the test results.  The harness is
 200  free, however, to display the diagnostics.  Typically diagnostics are
 201  used to provide information about the environment in which test file is
 202  running, or to delineate a group of tests.
 203  
 204      ...
 205      ok 18 - Closed database connection
 206      # End of database section.
 207      # This starts the network part of the test.
 208      # Daemon started on port 2112
 209      ok 19 - Opened socket
 210      ...
 211      ok 47 - Closed socket
 212      # End of network tests
 213  
 214  =head2 Anything else
 215  
 216  Any output line that is not a plan, a test line or a diagnostic is
 217  incorrect.  How a harness handles the incorrect line is undefined.
 218  Test::Harness silently ignores incorrect lines, but will become more
 219  stringent in the future.
 220  
 221  =head1 EXAMPLES
 222  
 223  All names, places, and events depicted in any example are wholly
 224  fictitious and bear no resemblance to, connection with, or relation to any
 225  real entity. Any such similarity is purely coincidental, unintentional,
 226  and unintended.
 227  
 228  =head2 Common with explanation
 229  
 230  The following TAP listing declares that six tests follow as well as
 231  provides handy feedback as to what the test is about to do. All six
 232  tests pass.
 233  
 234      1..6
 235      #
 236      # Create a new Board and Tile, then place
 237      # the Tile onto the board.
 238      #
 239      ok 1 - The object isa Board
 240      ok 2 - Board size is zero
 241      ok 3 - The object isa Tile
 242      ok 4 - Get possible places to put the Tile
 243      ok 5 - Placing the tile produces no error
 244      ok 6 - Board size is 1
 245  
 246  =head2 Unknown amount and failures
 247  
 248  This hypothetical test program ensures that a handful of servers are
 249  online and network-accessible. Because it retrieves the hypothetical
 250  servers from a database, it doesn't know exactly how many servers it
 251  will need to ping. Thus, the test count is declared at the bottom after
 252  all the test points have run. Also, two of the tests fail.
 253  
 254      ok 1 - retrieving servers from the database
 255      # need to ping 6 servers
 256      ok 2 - pinged diamond
 257      ok 3 - pinged ruby
 258      not ok 4 - pinged saphire
 259      ok 5 - pinged onyx
 260      not ok 6 - pinged quartz
 261      ok 7 - pinged gold
 262      1..7
 263  
 264  =head2 Giving up
 265  
 266  This listing reports that a pile of tests are going to be run. However,
 267  the first test fails, reportedly because a connection to the database
 268  could not be established. The program decided that continuing was
 269  pointless and exited.
 270  
 271      1..573
 272      not ok 1 - database handle
 273      Bail out! Couldn't connect to database.
 274  
 275  =head2 Skipping a few
 276  
 277  The following listing plans on running 5 tests. However, our program
 278  decided to not run tests 2 thru 5 at all. To properly report this,
 279  the tests are marked as being skipped.
 280  
 281      1..5
 282      ok 1 - approved operating system
 283      # $^0 is solaris
 284      ok 2 - # SKIP no /sys directory
 285      ok 3 - # SKIP no /sys directory
 286      ok 4 - # SKIP no /sys directory
 287      ok 5 - # SKIP no /sys directory
 288  
 289  =head2 Skipping everything
 290  
 291  This listing shows that the entire listing is a skip. No tests were run.
 292  
 293      1..0 # skip because English-to-French translator isn't installed
 294  
 295  =head2 Got spare tuits?
 296  
 297  The following example reports that four tests are run and the last two
 298  tests failed. However, because the failing tests are marked as things
 299  to do later, they are considered successes. Thus, a harness should report
 300  this entire listing as a success.
 301  
 302      1..4
 303      ok 1 - Creating test program
 304      ok 2 - Test program runs, no error
 305      not ok 3 - infinite loop # TODO halting problem unsolved
 306      not ok 4 - infinite loop 2 # TODO halting problem unsolved
 307  
 308  =head2 Creative liberties
 309  
 310  This listing shows an alternate output where the test numbers aren't
 311  provided. The test also reports the state of a ficticious board game in
 312  diagnostic form. Finally, the test count is reported at the end.
 313  
 314      ok - created Board
 315      ok
 316      ok
 317      ok
 318      ok
 319      ok
 320      ok
 321      ok
 322      # +------+------+------+------+
 323      # |      |16G   |      |05C   |
 324      # |      |G N C |      |C C G |
 325      # |      |  G   |      |  C  +|
 326      # +------+------+------+------+
 327      # |10C   |01G   |      |03C   |
 328      # |R N G |G A G |      |C C C |
 329      # |  R   |  G   |      |  C  +|
 330      # +------+------+------+------+
 331      # |      |01G   |17C   |00C   |
 332      # |      |G A G |G N R |R N R |
 333      # |      |  G   |  R   |  G   |
 334      # +------+------+------+------+
 335      ok - board has 7 tiles + starter tile
 336      1..9
 337  
 338  =head1 Non-Perl TAP
 339  
 340  In Perl, we use Test::Simple and Test::More to generate TAP output.
 341  Other languages have solutions that generate TAP, so that they can take
 342  advantage of Test::Harness.
 343  
 344  The following sections are provided by their maintainers, and may not
 345  be up-to-date.
 346  
 347  =head2 C/C++
 348  
 349  libtap makes it easy to write test programs in C that produce
 350  TAP-compatible output.  Modeled on the Test::More API, libtap contains
 351  all the functions you need to:
 352  
 353  =over 4
 354  
 355  =item * Specify a test plan
 356  
 357  =item * Run tests
 358  
 359  =item * Skip tests in certain situations
 360  
 361  =item * Have TODO tests
 362  
 363  =item * Produce TAP compatible diagnostics
 364  
 365  =back
 366  
 367  More information about libtap, including download links, checksums,
 368  anonymous access to the Subersion repository, and a bug tracking
 369  system, can be found at:
 370  
 371      http://jc.ngo.org.uk/trac-bin/trac.cgi/wiki/LibTap
 372  
 373  (Nik Clayton, April 17, 2006)
 374  
 375  =head2 Python
 376  
 377  PyTap will, when it's done, provide a simple, assertive (Test::More-like)
 378  interface for writing tests in Python.  It will output TAP and will
 379  include the functionality found in Test::Builder and Test::More.  It will
 380  try to make it easy to add more test code (so you can write your own
 381  C<TAP.StringDiff>, for example.
 382  
 383  Right now, it's got a fair bit of the basics needed to emulate Test::More,
 384  and I think it's easy to add more stuff -- just like Test::Builder,
 385  there's a singleton that you can get at easily.
 386  
 387  I need to better identify and finish implementing the most basic tests.
 388  I am not a Python guru, I just use it from time to time, so my aim may
 389  not be true.  I need to write tests for it, which means either relying
 390  on Perl for the tester tester, or writing one in Python.
 391  
 392  Here's a sample test, as found in my Subversion:
 393  
 394      from TAP.Simple import *
 395  
 396      plan(15)
 397  
 398      ok(1)
 399      ok(1, "everything is OK!")
 400      ok(0, "always fails")
 401  
 402      is_ok(10, 10, "is ten ten?")
 403      is_ok(ok, ok, "even ok is ok!")
 404      ok(id(ok),    "ok is not the null pointer")
 405      ok(True,      "the Truth will set you ok")
 406      ok(not False, "and nothing but the truth")
 407      ok(False,     "and we'll know if you lie to us")
 408  
 409      isa_ok(10, int, "10")
 410      isa_ok('ok', str, "some string")
 411  
 412      ok(0,    "zero is true", todo="be more like Ruby!")
 413      ok(None, "none is true", skip="not possible in this universe")
 414  
 415      eq_ok("not", "equal", "two strings are not equal");
 416  
 417  (Ricardo Signes, April 17, 2006)
 418  
 419  =head2 JavaScript
 420  
 421  Test.Simple looks and acts just like TAP, although in reality it's
 422  tracking test results in an object rather than scraping them from a
 423  print buffer.
 424  
 425      http://openjsan.org/doc/t/th/theory/Test/Simple/
 426  
 427  (David Wheeler, April 17, 2006)
 428  
 429  =head2 PHP
 430  
 431  All the big PHP players now produce TAP
 432  
 433  =over
 434  
 435  =item * phpt
 436  
 437  Outputs TAP by default as of the yet-to-be-released PEAR 1.5.0
 438  
 439      http://pear.php.net/PEAR
 440  
 441  =item * PHPUnit
 442  
 443  Has a TAP logger (since 2.3.4)
 444  
 445      http://www.phpunit.de/wiki/Main_Page
 446  
 447  =item * SimpleTest
 448  
 449  There's a third-party TAP reporting extension for SimpleTest
 450  
 451      http://www.digitalsandwich.com/archives/51-Updated-Simpletest+Apache-Test.html
 452  
 453  =item * Apache-Test
 454  
 455  Apache-Test's PHP writes TAP by default and includes the standalone
 456  test-more.php
 457  
 458      http://search.cpan.org/dist/Apache-Test/
 459  
 460  =back
 461  
 462  (Geoffrey Young, April 17, 2006)
 463  
 464  =head1 AUTHORS
 465  
 466  Andy Lester, based on the original Test::Harness documentation by Michael Schwern.
 467  
 468  =head1 ACKNOWLEDGEMENTS
 469  
 470  Thanks to
 471  Pete Krawczyk,
 472  Paul Johnson,
 473  Ian Langworth
 474  and Nik Clayton
 475  for help and contributions on this document.
 476  
 477  The basis for the TAP format was created by Larry Wall in the
 478  original test script for Perl 1.  Tim Bunce and Andreas Koenig
 479  developed it further with their modifications to Test::Harness.
 480  
 481  =head1 COPYRIGHT
 482  
 483  Copyright 2003-2005 by
 484  Michael G Schwern C<< <schwern@pobox.com> >>,
 485  Andy Lester C<< <andy@petdance.com> >>.
 486  
 487  This program is free software; you can redistribute it and/or
 488  modify it under the same terms as Perl itself.
 489  
 490  See L<http://www.perl.com/perl/misc/Artistic.html>.
 491  
 492  =cut