Bencher-Backend
view release on metacpan or search on metacpan
lib/Bencher/Backend.pm view on Meta::CPAN
Will try to load module `Bencher::Scenario::<NAME>` and expect to find a package
variable in the module called `$scenario` which should be a hashref containing
the scenario specification.
_
schema => 'perl::modname*',
cmdline_aliases => {m=>{}},
completion => sub { _complete_scenario_module(@_) },
tags => ['category:scenario', 'category:participant'],
},
cpanmodules_module => {
summary => 'Load a scenario from an Acme::CPANModules:: Perl module',
description => <<'_',
An <pm:Acme::CPANModules> module can also contain benchmarking information, e.g.
<pm:Acme::CPANModules::TextTable>.
_
schema => 'perl::modname*',
completion => sub { _complete_cpanmodules_module(@_) },
tags => ['category:scenario', 'category:participant'],
},
scenario => {
summary => 'Load a scenario from data structure',
schema => ['hash*'], # XXX bencher::scenario
tags => ['hidden-cli'],
tags => ['category:scenario', 'category:participant'],
},
participants => {
'summary' => 'Add participants',
schema => ['array*', of=>['hash*']],
cmdline_aliases => {
participant => $_alias_spec_add_participant,
p => $_alias_spec_add_participant,
},
tags => ['category:participant'],
},
datasets => {
summary => 'Add datasets',
schema => ['array*', of=>['hash*']],
cmdline_aliases => {
dataset => $_alias_spec_add_dataset,
d => $_alias_spec_add_dataset,
},
tags => ['category:dataset'],
},
env_hashes => {
summary => 'Add environment hashes',
schema => ['array*', of=>['hash*']],
cmdline_aliases => {
env_hash => $_alias_spec_add_env_hash,
},
},
precision => {
summary => 'Precision',
description => <<'_',
When benchmarking with the default <pm:Benchmark::Dumb> runner, will pass the
precision to it. The value is a fraction, e.g. 0.5 (for 5% precision), 0.01 (for
1% precision), and so on. Or, it can also be a positive integer to speciify
minimum number of iterations, usually need to be at least 6 to avoid the "Number
of initial runs is very small (<6)" warning. The default precision is 0, which
is to let Benchmark::Dumb determine the precision, which is good enough for most
cases.
When benchmarking with <pm:Benchmark> runner, will pass this value as the
C<$count> argument. Which can be a positive integer to mean the number of
iterations to do (e.g. 10, or 100). Or, can also be set to a negative number
(e.g. -0.5 or -2) to mean minimum number of CPU seconds. The default is -0.5.
When benchmarking with <pm:Benchmark::Dumb::SimpleTime>, this value is a
positive integer which means the number of iterations to perform.
When profiling, a number greater than 1 will set a repetition loop (e.g. C<<
for(1..100){ ... } >>).
This setting overrides `default_precision` property in the scenario.
_
schema => ['float*'],
},
precision_limit => {
summary => 'Set precision limit',
description => <<'_',
Instead of setting `precision` which forces a single value, you can also set
this `precision_limit` setting. If the precision in the scenario is higher
(=number is smaller) than this limit, then this limit is used. For example, if
the scenario specifies `default_precision` 0.001 and `precision_limit` is set to
0.005 then 0.005 is used.
This setting is useful on slower computers which might not be able to reach the
required precision before hitting maximum number of iterations.
_
schema => ['float*', between=>[0,1]],
},
action => {
schema => ['str*', {
in=>[qw/
list-perls
list-scenario-modules
show-scenario
list-participants
list-participant-modules
list-datasets
list-items
show-items-codes
show-items-results
show-items-results-sizes
show-items-outputs
dump-items
dump-parsed-scenario
profile
bench
/]
# list-functions
}],
default => 'bench',
cmdline_aliases => {
a => {},
list_perls => {
is_flag => 1,
summary => 'Shortcut for -a list-perls',
code => sub { $_[0]{action} = 'list-perls' },
},
list_scenario_modules => {
is_flag => 1,
summary => 'Shortcut for -a list-scenario-modules',
code => sub { $_[0]{action} = 'list-scenario-modules' },
},
L => {
is_flag => 1,
summary => 'Shortcut for -a list-scenario-modules',
code => sub { $_[0]{action} = 'list-scenario-modules' },
},
show_scenario => {
is_flag => 1,
summary => 'Shortcut for -a show-scenario',
code => sub { $_[0]{action} = 'show-scenario' },
},
list_participants => {
is_flag => 1,
summary => 'Shortcut for -a list-participants',
code => sub { $_[0]{action} = 'list-participants' },
},
list_participant_modules => {
is_flag => 1,
summary => 'Shortcut for -a list-participant-modules',
code => sub { $_[0]{action} = 'list-participant-modules' },
},
list_datasets => {
is_flag => 1,
lib/Bencher/Backend.pm view on Meta::CPAN
code => sub { $_[0]{action} = 'profile' },
},
},
tags => ['category:action'],
},
raw => {
summary => 'Show "raw" data',
schema => ['bool'],
description => <<'_',
When action=show-items-result, will print result as-is instead of dumping as
Perl.
_
tags => ['category:output'],
},
test => {
summary => 'Whether to test participant code once first before benchmarking',
schema => ['bool*'],
description => <<'_',
By default, participant code is run once first for testing (e.g. whether it dies
or return the correct result) before benchmarking. If your code runs for many
seconds, you might want to skip this test and set this to 0.
_
},
module_startup => {
schema => ['bool*', is=>1],
summary => 'Benchmark module startup overhead instead of normal benchmark',
tags => ['category:action'],
},
code_startup => {
schema => ['bool*', is=>1],
summary => 'Benchmark code startup overhead instead of normal benchmark',
tags => ['category:action'],
},
detail => {
summary => 'Show detailed information for each result',
schema => ['bool*'],
cmdline_aliases => {l=>{}},
},
runner => {
summary => 'Runner module to use',
schema => ['str*', {
in=>[
'Benchmark::Dumb',
'Benchmark',
'Benchmark::Dumb::SimpleTime',
],
}],
description => <<'_',
The default is `Benchmark::Dumb` which should be good enough for most cases.
You can use `Benchmark` runner (`Benchmark.pm`) if you are accustomed to it and
want to see its output format.
You can use `Benchmark::Dumb::SimpleTime` if your participant code runs for at
least a few to many seconds and you want to use very few iterations (like 1 or
2) because you don't want to wait for too long.
_
},
include_modules => {
'x.name.is_plural' => 1,
summary => 'Only include modules specified in this list',
'summary.alt.plurality.singular' => 'Add module to include list',
schema => ['array*', of=>['str*'], 'x.perl.coerce_rules' => ['From_str::comma_sep']],
element_completion => sub { _complete_participant_module(@_, apply_filters=>0) },
completion => sub { _complete_participant_modules_comma_sep(@_) },
tags => ['category:filtering'],
},
include_module_pattern => {
summary => 'Only include modules matching this regex pattern',
schema => ['re*'],
tags => ['category:filtering'],
},
exclude_modules => {
'x.name.is_plural' => 1,
summary => 'Exclude modules specified in this list',
'summary.alt.plurality.singular' => 'Add module to exclude list',
schema => ['array*', of=>['str*'], 'x.perl.coerce_rules' => ['From_str::comma_sep']],
element_completion => sub { _complete_participant_module(@_, apply_filters=>0) },
completion => sub { _complete_participant_modules_comma_sep(@_) },
tags => ['category:filtering'],
},
exclude_module_pattern => {
summary => 'Exclude module(s) matching this regex pattern',
schema => ['re*'],
tags => ['category:filtering'],
},
exclude_xs_modules => {
summary => 'Exclude XS modules',
schema => ['bool*', is=>1],
cmdline_aliases => { noxs => {is_flag=>1} },
tags => ['category:filtering'],
},
exclude_pp_modules => {
summary => 'Exclude PP (pure-Perl) modules',
schema => ['bool*', is=>1],
cmdline_aliases => { nopp => {is_flag=>1} },
tags => ['category:filtering'],
},
include_functions => {
'x.name.is_plural' => 1,
summary => 'Only include functions specified in this list',
'summary.alt.plurality.singular' => 'Add function to include list',
schema => ['array*', of=>['str*'], 'x.perl.coerce_rules' => ['From_str::comma_sep']],
element_completion => sub { _complete_function(@_, apply_filters=>0) },
completion => sub { _complete_functions_comma_sep(@_) },
tags => ['category:filtering'],
},
include_function_pattern => {
summary => 'Only include functions matching this regex pattern',
schema => ['re*'],
tags => ['category:filtering'],
lib/Bencher/Backend.pm view on Meta::CPAN
Benchmark multiple module versions.
If set to a module name, will search for all (instead of the first occurrence)
of the module in C<@INC>. Then will generate items for each version.
Currently only one module can be multi version.
=item * B<multiperl> => I<bool> (default: 0)
Benchmark against multiple perls.
Requires L<App::perlbrew> to be installed. Will use installed perls from the
perlbrew installation. Each installed perl must have L<Bencher::Backend>
module installed (in addition to having all modules that you want to benchmark,
obviously).
By default, only perls having Bencher::Backend will be included. Use
C<--include-perl> and C<--exclude-perl> to include and exclude which perls you
want.
Also note that due to the way this is currently implemented, benchmark code that
contains closures (references to variables outside the code) won't work.
=item * B<note> => I<str>
Put additional note in the result.
=item * B<on_failure> => I<str>
What to do when there is a failure.
For a command participant, failure means non-zero exit code. For a Perl-code
participant, failure means Perl code dies or (if expected result is specified)
the result is not equal to the expected result.
The default is "die". When set to "skip", will first run the code of each item
before benchmarking and trap command failure/Perl exception and if that happens,
will "skip" the item.
=item * B<on_result_failure> => I<str>
What to do when there is a result failure.
This is like C<on_failure> except that it specifically refer to the failure of
item's result not being equal to expected result.
There is an extra choice of C<warn> for this type of failure, which is to print a
warning to STDERR and continue.
=item * B<participants> => I<array[hash]>
Add participants.
=item * B<precision> => I<float>
Precision.
When benchmarking with the default L<Benchmark::Dumb> runner, will pass the
precision to it. The value is a fraction, e.g. 0.5 (for 5% precision), 0.01 (for
1% precision), and so on. Or, it can also be a positive integer to speciify
minimum number of iterations, usually need to be at least 6 to avoid the "Number
of initial runs is very small (<6)" warning. The default precision is 0, which
is to let Benchmark::Dumb determine the precision, which is good enough for most
cases.
When benchmarking with L<Benchmark> runner, will pass this value as the
C<$count> argument. Which can be a positive integer to mean the number of
iterations to do (e.g. 10, or 100). Or, can also be set to a negative number
(e.g. -0.5 or -2) to mean minimum number of CPU seconds. The default is -0.5.
When benchmarking with L<Benchmark::Dumb::SimpleTime>, this value is a
positive integer which means the number of iterations to perform.
When profiling, a number greater than 1 will set a repetition loop (e.g. C<<
for(1..100){ ... } >>).
This setting overrides C<default_precision> property in the scenario.
=item * B<precision_limit> => I<float>
Set precision limit.
Instead of setting C<precision> which forces a single value, you can also set
this C<precision_limit> setting. If the precision in the scenario is higher
(=number is smaller) than this limit, then this limit is used. For example, if
the scenario specifies C<default_precision> 0.001 and C<precision_limit> is set to
0.005 then 0.005 is used.
This setting is useful on slower computers which might not be able to reach the
required precision before hitting maximum number of iterations.
=item * B<raw> => I<bool>
Show "raw" data.
When action=show-items-result, will print result as-is instead of dumping as
Perl.
=item * B<render_as_benchmark_pm> => I<true>
Format result like Benchmark.pm.
=item * B<result_dir> => I<dirname>
Directory to use when saving benchmark result.
Default is from C<BENCHER_RESULT_DIR> environment variable, or the home
directory.
=item * B<result_filename> => I<filename>
Filename to use when saving benchmark result.
Default is:
<NAME>.<yyyy-dd-dd-"T"HH-MM-SS>.json
or, when running in module startup mode:
<NAME>.module_startup.<yyyy-dd-dd-"T"HH-MM-SS>.json
or, when running in code startup mode:
<NAME>.code_startup.<yyyy-dd-dd-"T"HH-MM-SS>.json
where <NAME> is scenario module name, or C<NO_MODULE> if scenario is not from a
module. The C<::> (double colon in the module name will be replaced with C<->
(dash).
=item * B<return_meta> => I<bool>
Whether to return extra metadata.
When set to true, will return extra metadata such as platform information, CPU
information, system load before & after the benchmark, system time, and so on.
This is put in result metadata under C<func.*> keys.
The default is to true (return extra metadata) unless when run as CLI and format
is text (where the extra metadata is not shown).
=item * B<runner> => I<str>
Runner module to use.
The default is C<Benchmark::Dumb> which should be good enough for most cases.
You can use C<Benchmark> runner (C<Benchmark.pm>) if you are accustomed to it and
want to see its output format.
You can use C<Benchmark::Dumb::SimpleTime> if your participant code runs for at
least a few to many seconds and you want to use very few iterations (like 1 or
2) because you don't want to wait for too long.
=item * B<save_result> => I<bool>
Whether to save benchmark result to file.
Will also be turned on automatically if C<BENCHER_RESULT_DIR> environment
variabl is defined.
When this is turned on, will save a JSON file after benchmark, containing the
result along with metadata. The directory of the JSON file will be determined
from the C<results_dir> option, while the filename from the C<results_filename>
option.
=item * B<scenario> => I<hash>
Load a scenario from data structure.
=item * B<scenario_file> => I<str>
Load a scenario from a Perl file.
Perl file will be do()'ed and the last expression should be a hash containing
the scenario specification.
=item * B<scenario_module> => I<perl::modname>
Load a scenario from a Bencher::Scenario:: Perl module.
Will try to load module C<< Bencher::Scenario::E<lt>NAMEE<gt> >> and expect to find a package
variable in the module called C<$scenario> which should be a hashref containing
the scenario specification.
=item * B<scientific_notation> => I<bool>
(No description)
=item * B<sorts> => I<array[str]> (default: ["-time"])
(No description)
=item * B<test> => I<bool>
Whether to test participant code once first before benchmarking.
By default, participant code is run once first for testing (e.g. whether it dies
or return the correct result) before benchmarking. If your code runs for many
seconds, you might want to skip this test and set this to 0.
=item * B<tidy> => I<bool>
Run perltidy over generated scripts.
=item * B<with_args_size> => I<bool>
Also return memory usage of item's arguments.
Memory size is measured using L<Devel::Size>.
=item * B<with_process_size> => I<bool>
( run in 1.561 second using v1.01-cache-2.11-cpan-96521ef73a4 )