view release on metacpan or  search on metacpan

lib/App/TestOnTap.pod  view on Meta::CPAN

=head1 NAME

App::TestOnTap - Test driver

Run a suite of test 'executables' in a directory tree, summarize
their TAP output and optionally save the aggregated results for later processing.

=head1 VERSION

Version 1.001

=head1 SYNOPSIS

  # Normally run 'testontap' from a shell prompt:
  #
  testontap --jobs 10 path/to/suite/root

=head1 OVERVIEW

B<NOTE: THIS IS CURRENTLY CONSIDERED TO BE IN BETA STATE. IT MAY CHANGE IN INCOMPATIBLE WAYS BEFORE THE 1.000 RELEASE.>

This package is an application, generally intended to be run from a command line.

The purpose of the application is to act as a "driver", in terms of automatically executing arbitrary test scripts/executables
(the "test suite") which in turn perform tests against whatever they're written to test (the "system under test", "SUT").
As such, the application has no idea what the tests actually do, it is only concerned about receiving TAP data back.

To help make the test suite comparatively easy to write and maintain, tests will be autodiscovered in it, based on a test suite
configuration, and also, using rules in this configuration, enable control of the execution order of the tests.

Finally, it can optionally aggregate all TAP data into a defined format (e.g. a directory tree or
directly to an archive) for later evaluation or other handling elsewhere (e.g. into a database with a custom UI on top).  

So, summarizing this into a simple list of general requirements:

=over

=item * B<Scalable and easy to create small or large test suites.>

Test suites could be small, developer oriented with just a few basic smoke-test scripts for quick system- or unit-testing during
development, all the way to an enormous number of executables in a test suite for lengthy and in depth system-testing of a deployed system.
Therefore the test suite should be easy to set up as an arbitrarily complex directory structure.

Preferably, writing a new test should require focusing almost entirely on the complexities of the test itself (which can certainly
have its own challenges!) rather than how it should fit into the suite, at least at first. Thus, it's beneficial if there by default
is no particular administration around a given test - just create the test script and it's automatically found and run.

=item * B<Allow tests to run in a defined order, including when running in parallel.>

In many cases, the time it takes to run a test suite is pretty much irrelevant, perhaps because it's fairly seldomly used and run only at night
or simply because the tests are few and/or very quick. In these situations, it's ok to have the test driver run the tests in basically
any order (currently, simple alphabetic order is used).

However, it may turn out that tests can be parallelized. In the best case, no test is dependent on another, so any amount of parallelization
will work (given what a specific host can comfortably handle). In more complex cases, tests may depend on others in various ways so a
mechanism is needed to express such dependencies, allowing the driver to schedule test scripts appropriately whether the tests are run
one at a time or using 'n' possible parallel slots.  

=item * B<Handle result aggregation for entire test suite, with optional persistence of this data.>

In a typical smoke-test test suite used during development, it's probably enough to just rely on eyeballing the test result, so it's
enough with simple console based output on which tests worked and which ones failed (including which individual micro-tests failed).

For something like automated daily system tests however, it's typically important to retain the results for perusal later. In more
advanced scenarios, such persisted results could be used to build a history of test results, for example measuring trends over the past
months, creating graphs of data or similar. For this reason, it is useful if the test results can be packaged in a practical format, e.g.
as an archive.
 
=item * B<Very loose coupling between this application and test suite, system under test and other variables.>

The implementation of the driver should have very little to do with how the tests or the SUT is implemented, or what or how
they perform their stuff. 

=item * B<Language and platform agnostic.>