view release on metacpan or  search on metacpan

lib/Benchmark/Lab.pm  view on Meta::CPAN

=item *

when profiling, benchmarking stops after minimums are satisfied

=item *

when not profiling, benchmarking stops once one of C<max_secs> or C<max_reps> is exceeded.

=back

Note that "elapsed time" for the C<min_secs> and C<max_secs> is wall-clock
time, not the cumulative recorded time of the task itself.

=head2 start

    my $result = $bm->start( $package, $context, $label );

This method executes the structured benchmark from the given C<$package>.
The C<$context> parameter is passed to all task phases.  The C<$label>
is used for diagnostic output to describe the benchmark being run.

If parameters are omitted, C<$package> defaults to "main", an empty
hash reference is used for the C<$context>, and the C<$label> defaults
to the C<$package>.

It returns a hash reference with the following keys:

=over 4

=item *

C<elapsed> – total wall clock time to execute the benchmark (including non-timed portions).

=item *

C<total_time> – sum of recorded task iterations times.

=item *

C<iterations> – total number of C<do_task> functions called.

=item *

C<percentiles> – hash reference with 1, 5, 10, 25, 50, 75, 90, 95 and 99th percentile iteration times.  There may be duplicates if there were fewer than 100 iterations.

=item *

C<median_rate> – the inverse of the 50th percentile time.

=item *

C<timing> – array reference with individual iteration times as (floating point) seconds.

=back

=for Pod::Coverage BUILD

=head1 CAVEATS

If the C<do_task> executes in less time than the timer granularity, an
error will be thrown.  For benchmarks that do not have before/after functions,
just repeating the function under test in C<do_task> will be sufficient.

=head1 RATIONALE

I believe most approaches to benchmarking are flawed, primarily because
they focus on finding a I<single> measurement.  Single metrics are easy to
grok and easy to compare ("foo was 13% faster than bar!"), but they obscure
the full distribution of timing data and (as a result) are often unstable.

Most of the time, people hand-wave this issue and claim that the Central
Limit Theorem (CLT) solves the problem for a large enough sample size.
Unfortunately, the CLT holds only if means and variances are finite and
some real world distributions are not (e.g. hard drive error frequencies
best fit a Pareto distribution).

Further, we often care more about the shape of the distribution than just a
single point.  For example, I would rather have a process with mean µ that
stays within 0.9µ - 1.1µ  than one that varies from 0.5µ - 1.5µ.

And a process that is 0.1µ 90% of the time and 9.1µ 10% of the time (still
with mean µ!) might be great or terrible, depending on the application.

This module grew out of a desire for detailed benchmark timing data, plus
some additional features, which I couldn't find in existing benchmarking
modules:

=over 4

=item *

Raw timing data – I wanted to be able to get raw timing data, to allow more flexible statistical analysis of timing distributions.

=item *

Monotonic clock – I wanted times from a high-resolution monotonic clock (if available).

=item *

Setup/before/after/teardown – I wanted to be able to initialize/reset state not just once at the start, but before each iteration and without it being timed.

=item *

L<Devel::NYTProf> integration – I wanted to be able to run the B<exact> same code I benchmarked through L<Devel::NYTProf>, also limiting the profiler to the benchmark task alone, not the setup/teardown/etc. code.

=back

Eventually, I hope to add some more robust graphic visualization and
statistical analyses of timing distributions.  This might include both
single-point estimates (like other benchmarking modules) but also more
sophisticated metrics, like non-parametric measures for comparing samples
with unequal variances.

=head1 SEE ALSO

There are many benchmarking modules on CPAN with a mix of features that
may be sufficient for your needs.  To my knowledge, none give timing
distributions or integrate with L<Devel::NYTProf>.

Here is a brief rundown of some that I am familiar with: