view release on metacpan or  search on metacpan

lib/Benchmark/DKbench.pm  view on Meta::CPAN

here are some general guidelines for various systems.

=head2 Linux / WSL etc

The only non-CPAN software required to install/run the suite is a build environment
for the C/XS modules (C compiler, make etc.) and Perl. On the most popular Linux
package managers you can easily set up such an environment (as root or with sudo):

 # Debian/Ubuntu etc
 apt-get update
 apt-get install build-essential perl cpanminus

 # CentOS/Red Hat
 yum update
 yum install gcc make patch perl perl-App-cpanminus

After that, you can use L<App::cpanminus> to install the benchmark suite (as
root/sudo is the easiest, will install for all users):

 cpanm -n Benchmark::DKbench

=head2 Solaris

You will need to install the Oracle Solaris Studio development package to have a
compiler environment, and to add its C<bin> directory to your PATH, before installing
the benchmark suite.

=head2 Strawberry Perl

If you are on Windows, you should be using the Windows Subsystem for Linux (WSL)
for running Perl or, if you can't (e.g. old Windows versions), Cygwin instead.
The suite should still work on Strawberry Perl, as long as you don't try to run
tests when installing (some dependencies will not pass them). The simplest way is
with L<App::cpanminus> (most Strawberry Perl versions have it installed):

 cpanm -n Benchmark::DKbench

otherwise with the base CPAN shell:

 perl -MCPAN -e shell

 > notest install Benchmark::DKbench

and then note that the scripts get the batch extension appended, so C<dkbench.bat>
runs the suite (and C<setup_dkbench.bat> can assist with module versions, optional
benchmarks etc.).

Be aware that Strawberry Perl is slower, on my test system I get almost 50% slower
performance than WSL and 30% slower than Cygwin.

=head1 SCRIPTS

You will most likely only ever need the main script C<dkbench> which launches the
suite, although C<setup_dkbench> can help with setup or standardizing/normalizing your
benchmarking environment.

=head2 C<dkbench>

The main script that runs the DKbench benchmark suite. If L<BioPerl> is installed,
you may want to start with C<dkbench --setup>. But beyond that, there are many
options to control number of threads, iterations, which benchmarks to run etc:

 dkbench [options]

 Options:
 --threads <i>,  -j <i> : Number of benchmark threads (default is 1).
 --multi,        -m     : Multi-threaded using all your CPU cores/threads.
 --max_threads <i>      : Override the CPU detection to specify max CPU threads.
 --iter <i>,     -i <i> : Number of suite iterations (with min/max/avg at the end).
 --stdev                : Show relative standard deviation (for iter > 1).
 --include <regex>      : Run only benchmarks that match regex.
 --exclude <regex>      : Do not run benchmarks that match regex.
 --time,         -t     : Report time (sec) instead of score.
 --quick,        -q     : Quick benchmark run (implies -t).
 --no_mce               : Do not run under MCE::Loop (implies -j 1).
 --scale <i>,    -s <i> : Scale the bench workload by x times (incompatible with -q).
 --skip_bio             : Skip BioPerl benchmarks.
 --skip_prove           : Skip Moose prove benchmark.
 --time_piece           : Run optional Time::Piece benchmark (see benchmark details).
 --bio_codons           : Run optional BioPerl Codons benchmark (does not scale well).
 --sleep <i>            : Sleep for <i> secs after each benchmark.
 --duration <i>, -d <i> : Minimum duration in seconds for suite run.
 --setup                : Download the GenBank data to enable the BioPerl tests.
 --datapath <path>      : Override the path where the expected benchmark data is found.
 --ver <num>            : Skip benchmarks added after the specified version.
 --help          -h     : Show basic help and exit.

The default run (no options) will run all the benchmarks both single-threaded and
multi-threaded (using all detected CPU cores/hyper-threads) and show you scores and
multi vs single threaded scalability.

The scores are calibrated such that a reference CPU (Intel Xeon Platinum 8481C -
Sapphire Rapids) would achieve a score of 1000 in a single-core benchmark run using
the default software configuration (Linux/Perl 5.36+ built with multiplicity and
threads, with reference CPAN module versions). Perl built without thread support and
multi(plicity) will be a bit faster (usually in the order of ~3-4%), while older Perl
versions will most likely be slower. Different CPAN module versions will also impact
scores, using C<setup_dkbench> is a way to ensure a reference environment for more
meaningful hardware comparisons.

The multi-thread scalability calculated by the suite should approach 100% if each
thread runs on a full core (i.e. no SMT), and the core can maintain the clock speed
it had on the single-thread runs. Note that the overall scalability is an average
of the benchmarks that drops non-scaling outliers (over 2*stdev less than the mean).

If you want to reduce the effects of thermal throttling, which will lower the speed
of (mainly multi-threaded) benchmarks as the CPU temperature increases, the C<sleep>
option can help by adding cooldown time between each benchmark.

The suite will report a Pass/Fail per benchmark. A failure may be caused if you have
different CPAN module version installed - this is normal, and you will be warned.

L<MCE::Loop> is used to run on the desired number of parallel threads, with minimal
overhead. There is an option to disable it, which forces a single-thread run.

=head2 C<setup_dkbench>

Simple installer to check/get the reference versions of CPAN modules and download
the GenBank data file required for the BioPerl benchmarks of the DKbench suite.

It assumes that you have some software already installed (see L</"INSTALLATION"> above),
try C<setup_dkbench --help> will give you more details.

 setup_dkbench [--force --sudo --test --data=s --help]

 Options:
 --sudo   : Will use sudo for cpanm calls.
 --force  : Will install reference CPAN module versions and re-download the genbank data.
 --test   : Will run the test suites for the CPAN module (default behaviour is to skip).