view release on metacpan or  search on metacpan

ext/zstd/tests/README.md  view on Meta::CPAN

Programs and scripts for automated testing of Zstandard
=======================================================

This directory contains the following programs and scripts:
- `datagen` : Synthetic and parametrable data generator, for tests
- `fullbench`  : Precisely measure speed for each zstd inner functions
- `fuzzer`  : Test tool, to check zstd integrity on target platform
- `paramgrill` : parameter tester for zstd
- `test-zstd-speed.py` : script for testing zstd speed difference between commits
- `test-zstd-versions.py` : compatibility test between zstd versions stored on Github (v0.1+)
- `zstreamtest` : Fuzzer test tool for zstd streaming API
- `legacy` : Test tool to test decoding of legacy zstd frames
- `decodecorpus` : Tool to generate valid Zstandard frames, for verifying decoder implementations


#### `test-zstd-versions.py` - script for testing zstd interoperability between versions

This script creates `versionsTest` directory to which zstd repository is cloned.
Then all tagged (released) versions of zstd are compiled.
In the following step interoperability between zstd versions is checked.

#### `automated-benchmarking.py` - script for benchmarking zstd prs to dev

This script benchmarks facebook:dev and changes from pull requests made to zstd and compares
them against facebook:dev to detect regressions. This script currently runs on a dedicated
desktop machine for every pull request that is made to the zstd repo but can also
be run on any machine via the command line interface.

There are three modes of usage for this script: fastmode will just run a minimal single
build comparison (between facebook:dev and facebook:release), onetime will pull all the current
pull requests from the zstd repo and compare facebook:dev to all of them once, continuous
will continuously get pull requests from the zstd repo and run benchmarks against facebook:dev.

```
Example usage: python automated_benchmarking.py
```

```
usage: automated_benchmarking.py [-h] [--directory DIRECTORY]
                                 [--levels LEVELS] [--iterations ITERATIONS]
                                 [--emails EMAILS] [--frequency FREQUENCY]
                                 [--mode MODE] [--dict DICT]

optional arguments:
  -h, --help            show this help message and exit
  --directory DIRECTORY
                        directory with files to benchmark
  --levels LEVELS       levels to test e.g. ('1,2,3')
  --iterations ITERATIONS
                        number of benchmark iterations to run
  --emails EMAILS       email addresses of people who will be alerted upon
                        regression. Only for continuous mode
  --frequency FREQUENCY
                        specifies the number of seconds to wait before each
                        successive check for new PRs in continuous mode
  --mode MODE           'fastmode', 'onetime', 'current', or 'continuous' (see
                        README.md for details)
  --dict DICT           filename of dictionary to use (when set, this
                        dictionary will be used to compress the files provided
                        inside --directory)
```

#### `test-zstd-speed.py` - script for testing zstd speed difference between commits

DEPRECATED

This script creates `speedTest` directory to which zstd repository is cloned.
Then it compiles all branches of zstd and performs a speed benchmark for a given list of files (the `testFileNames` parameter).
After `sleepTime` (an optional parameter, default 300 seconds) seconds the script checks repository for new commits.
If a new commit is found it is compiled and a speed benchmark for this commit is performed.
The results of the speed benchmark are compared to the previous results.
If compression or decompression speed for one of zstd levels is lower than `lowerLimit` (an optional parameter, default 0.98) the speed benchmark is restarted.
If second results are also lower than `lowerLimit` the warning e-mail is sent to recipients from the list (the `emails` parameter).

Additional remarks:
- To be sure that speed results are accurate the script should be run on a "stable" target system with no other jobs running in parallel
- Using the script with virtual machines can lead to large variations of speed results
- The speed benchmark is not performed until computers' load average is lower than `maxLoadAvg` (an optional parameter, default 0.75)
- The script sends e-mails using `mutt`; if `mutt` is not available it sends e-mails without attachments using `mail`; if both are not available it only prints a warning


The example usage with two test files, one e-mail address, and with an additional message:
```
./test-zstd-speed.py "silesia.tar calgary.tar" "email@gmail.com" --message "tested on my laptop" --sleepTime 60
```