Dumbbench

 view release on metacpan or  search on metacpan

bin/dumbbench  view on Meta::CPAN

Options:
 -p=X
 --precision=X     Set the target precision (default: 0.10=10%)
                   Set to 0 to disable.
 -a=x
 --absprecision=X  Set the target absolute precision (default: 0)
                   Set to 0 to disable.
 -v|--verbose      Increase verbosity. Increases up to three times.
 -i=X|--initial=X  Set number of initial timing runs (default: 20)
                   Increase, not decrease this number if possible.
 -m=X|--maxiter=X  Set a hard maximum number of iterations (default:1000)
                   If this hard limit is hit, the precision is off.
 -d=X|--dry-run=X  Set explicit dry-run command or code.
 --no-dry-run      Disable subtraction of dry runs.
 --raw             Set raw output mode. Only the final count will be
                   printed to stdout.
 --float           Numbers will be printed in
                   default float format instead of scientific notation.
 -s|--std          Use the standard deviation instead of the MAD as a
                   measure of variability.
 --code='code'     Benchmarks Perl code (can be specified multiple times

bin/dumbbench  view on Meta::CPAN

  my @discarded = Capture::Tiny::capture(sub {
    SOOT::Init(1);
  });
}

my $bench = Dumbbench->new(
  verbosity            => $V,
  target_rel_precision => $RelPrecision,
  target_abs_precision => $AbsPrecision,
  initial_runs         => $InitialTimings,
  max_iterations       => $MaxIter,
  variability_measure  => ($UseStdDeviation ? 'std_dev' : 'mad_dev' ),
  subtract_dry_run     => !$NoDryRun,
);

if (@CMD) {
  $bench->add_instances(
    Dumbbench::Instance::Cmd->new(
      name    => 'cmd',
      (
        defined $DryRunCmd

lib/Benchmark/Dumb.pm  view on Meta::CPAN

=item *

B<The C<$count> parameter is interpreted very differently!>

With C<Benchmark.pm>, specifying a positive integer meant that the
benchmark should be run exactly C<$count> times. A negative value
indicated that the code should be run until C<$count> seconds of
cumulated run-time have elapsed.

With C<Benchmark::Dumb>, we can do better. A positive integer
specifies the I<minimum> number of iterations. C<Dumbbench> may choose
to run more iterations to arrive at the necessary precision.

Specifying a certain target run-time (via a negative number for C<$count>)
may seem like a tempting idea, but if you care at all about the precision
of your result, it's quite useless.
B<This usage is not supported by C<Benchmark::Dumb>!>

Instead, if you pass a positive floating point number as C<$count>,
the fractional part of the number willbe interpreted as the target
I<relative precision> that you expect from the result.

lib/Benchmark/Dumb.pm  view on Meta::CPAN

These functions work mostly (see the C<$count> gotcha above)
like the equivalent functions in the C<Benchmark> module, but
the textual output is different in that it contains estimates
of the uncertainties. Some of the I<style> and I<format>
options of the original functions are ignored for the time being.

I'm quoting the C<Benchmark> documentation liberally.

=head2 timethis(COUNT, CODE, [TITLE, [STYLE]])

Time I<COUNT> iterations of I<CODE>. I<CODE> may be a string to eval
or a code reference. Unlike with the original C<Benchmark>,
the code will B<not> run in the caller's package. Results will be printed
to C<STDOUT> as I<TITLE> followed by the C<timestr()>.
I<TITLE> defaults to C<"timethis COUNT"> if none is provided.

I<STYLE> determines the format of the output, as described for C<timestr()> below.

Please read the section on C<Differences to Benchmark.pm>
for a discussion of how the I<COUNT> parameter is interpreted.

lib/Dumbbench.pm  view on Meta::CPAN

require Dumbbench::Stats;
require Dumbbench::Instance;

use Params::Util '_INSTANCE';

use Class::XSAccessor {
  getters => [qw(
    target_rel_precision
    target_abs_precision
    initial_runs
    max_iterations
    variability_measure
    started
    outlier_rejection
    subtract_dry_run
  )],
  accessors => [qw(verbosity)],
};


sub new {
  my $proto = shift;
  my $class = ref($proto)||$proto;
  my $self;
  if (not ref($proto)) {
    $self = bless {
      verbosity            => 0,
      target_rel_precision => 0.05,
      target_abs_precision => 0,
      initial_runs         => 20,
      max_iterations       => 10000,
      variability_measure  => 'mad',
      instances            => [],
      started              => 0,
      outlier_rejection    => 3,
      subtract_dry_run     => 1,
      @_,
    } => $class;
  }
  else {
    $self = bless {%$proto, @_} => $class;

lib/Dumbbench.pm  view on Meta::CPAN

  my $instance = shift;
  my $dry = shift;

  my $name = $instance->_name_prefix;

  # for overriding in case of dry-run mode
  my $V = $self->verbosity || 0;
  my $initial_timings = $self->initial_runs;
  my $abs_precision = $self->target_abs_precision;
  my $rel_precision = $self->target_rel_precision;
  my $max_iterations = $self->max_iterations;

  if ($dry) {
    $V--; $V = 0 if $V < 0;
    $initial_timings *= 5;
    $abs_precision    = 0;
    $rel_precision   /= 2;
    $max_iterations  *= 10;
  }

  print "${name}Running initial timing for warming up the cache...\n" if $V;
  if ($dry) {
    # be generous, this is fast
    $instance->single_dry_run() for 1..3;
  }
  else {
    $instance->single_run();
  }

lib/Dumbbench.pm  view on Meta::CPAN

  my $n_good = 0;
  my $variability_measure = $self->variability_measure;
  while (1) {
    my ($good, $outliers) = $stats->filter_outliers(
      variability_measure => $variability_measure,
      nsigma_outliers     => $self->outlier_rejection,
    );

    $n_good = @$good;

    if (not $n_good and @timings >= $max_iterations) {
      $mean = 0; $sigma = 0;
      last;
    }

    if ($n_good) {
      my $new_stats = Dumbbench::Stats->new(data => $good);
      $sigma = $new_stats->$variability_measure() / sqrt($n_good);
      $mean = $new_stats->mean();

      # stop condition

lib/Dumbbench.pm  view on Meta::CPAN

        print "${name}Reached relative precision $rel (neeed $rel_precision).\n" if $V > 1;
        $need_iter++ if $rel > $rel_precision;
      }
      if ($abs_precision > 0) {
        print "${name}Reached absolute precision $sigma (neeed $abs_precision).\n" if $V > 1;
        $need_iter++ if $sigma > $abs_precision;
      }
      if ($n_good < $initial_timings) {
        $need_iter++;
      }
      last if not $need_iter or @timings >= $max_iterations;
    }

    # progressively run more new timings in one go. Otherwise,
    # we start to stall on the O(n*log(n)) complexity of the median.
    my $n = List::Util::min( $max_iterations - @timings, List::Util::max(1, @timings*0.05) );
    push @timings, ($dry ? $instance->single_dry_run() : $instance->single_run()) for 1..$n;
  } # end while more data required

  if (@timings >= $max_iterations and not $dry) {
    print "${name}Reached maximum number of iterations. Stopping. Precision not reached.\n";
  }

  # rescale sigma
  # This is necessary since by cutting everything outside of n-sigma,
  # we artificially reduce the variability of the main distribution.
  if ($self->outlier_rejection) {
    # TODO implement
  }

  my $result = Dumbbench::Result->new(

lib/Dumbbench.pm  view on Meta::CPAN

  foreach my $instance ($self->instances) {
    my $result = $instance->result;
    my $result_str = ($options->{float}) ? unscientific_notation($result) : "$result";

    if (not $raw) {
      my $mean = $result->raw_number;
      my $sigma = $result->raw_error->[0];
      my $name = $instance->_name_prefix;
      printf(
        "%sRan %u iterations (%u outliers).\n",
        $name,
        scalar(@{$instance->timings}),
        scalar(@{$instance->timings})-$result->nsamples
      );
      printf(
        "%sRounded run time per iteration (seconds): %s (%.1f%%)\n",
        $name,
        $result_str,
        $sigma/$mean*100
      );

lib/Dumbbench.pm  view on Meta::CPAN

Dumbbench - More reliable benchmarking with the least amount of thinking

=head1 SYNOPSIS

Command line interface: (See C<dumbbench --help>)

  dumbbench -p 0.005 -- ./testprogram --testprogramoption

This will start churning for a while and then prints something like:

  Ran 23 iterations of the command.
  Rejected 3 samples as outliers.
  Rounded run time per iteration (seconds): 9.519e-01 +/- 3.7e-03 (0.4%)

As a module:

  use Dumbbench;

  my $bench = Dumbbench->new(
    target_rel_precision => 0.005, # seek ~0.5%
    initial_runs         => 20,    # the higher the more reliable

lib/Dumbbench.pm  view on Meta::CPAN

(which are also object attributes).

=head2 new

Constructor that takes the following arguments (with defaults):

  verbosity            => 0,     # 0, 1, or 2
  target_rel_precision => 0.05,  # 5% target precision
  target_abs_precision => 0,     # no target absolute precision (in s)
  intial_runs          => 20,    # no. of guaranteed initial runs
  max_iterations       => 10000, # hard max. no of iterations
  variability_measure  => 'mad', # method for calculating uncertainty
  outlier_rejection    => 3,     # no. of "sigma"s for the outlier rejection

C<variability_measure> and C<outlier_rejection> probably make sense
after reading C<HOW IT WORKS> below. Setting C<outlier_rejection> to 0
will turn it off entirely.

=head2 add_instances

Takes one or more instances of subclasses of L<Dumbbench::Instance>

simulator/lib/Dumbbench/Sim.pm  view on Meta::CPAN

  constructor => 'new',
  accessors   => [qw(
    config
    stats
    stats_good
    stats_outliers
    outlier_distribution
    data_distribution
    ev_before
    ev_after
    iterations_per_run
  )],
};

sub from_yaml {
  my $class = shift;
  my $file = shift;

  my $self = $class->new(
    config => Dumbbench::Sim::Config->from_yaml($file)
  );

simulator/lib/Dumbbench/Sim.pm  view on Meta::CPAN

# simulates one run
sub _sim_single_timing {
  my $self = shift;
  my %opts = @_;
  my $cfg = $self->config;

  # The logic behind $opts{setup}, the while(1) loop and $n is that Dumbbench will
  # first run a training run of your benchmark. If the run time is below some value
  # (currently 1.e-4), then it puts a loop around your code and keeps doubling
  # the loop count until the total time is above the limit.
  # THEN, it keeps that N fixed for the real iterations. Here, $opts{setup} indicates
  # the test run.

  my $time = 0;
  my $n = 1;
  my $iloop = 0;
  while (1) {
    # Simulate n levels of offsets with a probability of occurring $outlier_fraction^n
    my $offset = 0;
    while (rand() < $cfg->outlier_fraction) {
      $offset += $self->outlier_distribution->GetRandom()

simulator/lib/Dumbbench/Sim.pm  view on Meta::CPAN

    # end condition
    if ($iloop == $n) {
      # increase $n until we're above the limit
      if ($opts{setup}) {
        $n *= 2, next if $time < $cfg->duration_lower_limit;
      }
      last;
    }
  }

  # save the required no. of iterations for the actual runs
  if ($opts{setup}) {
    $self->iterations_per_run($n);
    print "Detected a required $n iterations per run.\n";
  }


  # discretization
  # In a nutshell, there is a limited precision to the clock
  # we're using. There is a feature to wait for the next clock
  # tick and then start measuring.
  # If we use this, then we always start at the beginning of the tick.
  # The reported time will be shorter than the actual time because
  # calling time() between ticks reports the time from the preceding tick