Compress-Zstd
view release on metacpan or search on metacpan
0.13 2018-04-01T03:50:50Z
- zstd 1.3.4
0.12 2018-03-19T17:28:58Z
- zstd 1.3.3
0.11 2017-10-28T09:58:01Z
- zstd 1.3.2
0.10 2017-09-30T07:13:12Z
- Add streaming interfaces
0.06 2017-09-24T12:23:22Z
- zstd 1.3.1
- Support >= Perl 5.26.0 (@INC problem)
- Support *BSD
0.05 2017-03-18T08:02:21Z
- zstd 1.1.4
0.04 2017-03-11T09:32:56Z
lib/Compress/Zstd/CompressionContext.pm
lib/Compress/Zstd/CompressionDictionary.pm
lib/Compress/Zstd/Compressor.pm
lib/Compress/Zstd/DecompressionContext.pm
lib/Compress/Zstd/DecompressionDictionary.pm
lib/Compress/Zstd/Decompressor.pm
minil.toml
ppport.h
t/00_compile.t
t/01_basic.t
t/02_streaming.t
t/03_context.t
t/04_dictionary.t
t/test.dic
typemap
xt/01_leaktrace.t
ext/zstd/CHANGELOG
ext/zstd/CODE_OF_CONDUCT.md
ext/zstd/CONTRIBUTING.md
ext/zstd/COPYING
ext/zstd/LICENSE
ext/zstd/doc/images/zstd_cdict_v1_3_5.png
ext/zstd/doc/images/zstd_logo86.png
ext/zstd/doc/zstd_compression_format.md
ext/zstd/doc/zstd_manual.html
ext/zstd/examples/Makefile
ext/zstd/examples/README.md
ext/zstd/examples/common.h
ext/zstd/examples/dictionary_compression.c
ext/zstd/examples/dictionary_decompression.c
ext/zstd/examples/multiple_simple_compression.c
ext/zstd/examples/multiple_streaming_compression.c
ext/zstd/examples/simple_compression.c
ext/zstd/examples/simple_decompression.c
ext/zstd/examples/streaming_compression.c
ext/zstd/examples/streaming_decompression.c
ext/zstd/examples/streaming_memory_usage.c
ext/zstd/lib/BUCK
ext/zstd/lib/Makefile
ext/zstd/lib/README.md
ext/zstd/lib/common/bitstream.h
ext/zstd/lib/common/compiler.h
ext/zstd/lib/common/cpu.h
ext/zstd/lib/common/debug.c
ext/zstd/lib/common/debug.h
ext/zstd/lib/common/entropy_common.c
ext/zstd/lib/common/error_private.c
ext/zstd/CHANGELOG view on Meta::CPAN
perf: Improve compression ratio for small windowLog by @cyan4973 (#1624)
perf: Faster compression speed in high compression mode for repetitive data by @terrelln (#1635)
api: Add parameter to generate smaller dictionaries by @tyler-tran (#1656)
cli: Recognize symlinks when built in C99 mode by @felixhandte (#1640)
cli: Expose cpu load indicator for each file on -vv mode by @ephiepark (#1631)
cli: Restrict read permissions on destination files by @chungy (#1644)
cli: zstdgrep: handle -f flag by @felixhandte (#1618)
cli: zstdcat: follow symlinks by @vejnar (#1604)
doc: Remove extra size limit on compressed blocks by @felixhandte (#1689)
doc: Fix typo by @yk-tanigawa (#1633)
doc: Improve documentation on streaming buffer sizes by @cyan4973 (#1629)
build: CMake: support building with LZ4 @leeyoung624 (#1626)
build: CMake: install zstdless and zstdgrep by @leeyoung624 (#1647)
build: CMake: respect existing uninstall target by @j301scott (#1619)
build: Make: skip multithread tests when built without support by @michaelforney (#1620)
build: Make: Fix examples/ test target by @sjnam (#1603)
build: Meson: rename options out of deprecated namespace by @lzutao (#1665)
build: Meson: fix build by @lzutao (#1602)
build: Visual Studio: don't export symbols in static lib by @scharan (#1650)
build: Visual Studio: fix linking by @absotively (#1639)
build: Fix MinGW-W64 build by @myzhang1029 (#1600)
ext/zstd/CHANGELOG view on Meta::CPAN
api: Fix ZSTD_decompressDCtx() corner cases with a dictionary
api: Move ZSTD_getDictID_*() functions to the stable section
api: Add ZSTD_c_literalCompressionMode flag to enable or disable literal compression by @terrelln
api: Allow compression parameters to be set when a dictionary is used
api: Allow setting parameters before or after ZSTD_CCtx_loadDictionary() is called
api: Fix ZSTD_estimateCStreamSize_usingCCtxParams()
api: Setting ZSTD_d_maxWindowLog to 0 means use the default
cli: Ensure that a dictionary is not used to compress itself by @shakeelrao
cli: Add --[no-]compress-literals flag to enable or disable literal compression
doc: Update the examples to use the advanced API
doc: Explain how to transition from old streaming functions to the advanced API in the header
build: Improve the Windows release packages
build: Improve CMake build by @hjmjohnson
build: Build fixes for FreeBSD by @lwhsu
build: Remove redundant warnings by @thatsafunnyname
build: Fix tests on OpenBSD by @bket
build: Extend fuzzer build system to work with the new clang engine
build: CMake now creates the libzstd.so.1 symlink
build: Improve Menson build by @lzutao
misc: Fix symbolic link detection on FreeBSD
misc: Use physical core count for -T0 on FreeBSD by @cemeyer
ext/zstd/CHANGELOG view on Meta::CPAN
misc: all /contrib projects fixed
misc: added /contrib/docker script by @gyscos
v1.3.3
perf: faster zstd_opt strategy (levels 16-19)
fix : bug #944 : multithreading with shared ditionary and large data, reported by @gsliepen
cli : fix : content size written in header by default
cli : fix : improved LZ4 format support, by @felixhandte
cli : new : hidden command `-S`, to benchmark multiple files while generating one result per file
api : fix : support large skippable frames, by @terrelln
api : fix : streaming interface was adding a useless 3-bytes null block to small frames
api : change : when setting `pledgedSrcSize`, use `ZSTD_CONTENTSIZE_UNKNOWN` macro value to mean "unknown"
build: fix : compilation under rhel6 and centos6, reported by @pixelb
build: added `check` target
v1.3.2
new : long range mode, using --long command, by Stella Lau (@stellamplau)
new : ability to generate and decode magicless frames (#591)
changed : maximum nb of threads reduced to 200, to avoid address space exhaustion in 32-bits mode
fix : multi-threading compression works with custom allocators
fix : ZSTD_sizeof_CStream() was over-evaluating memory usage
ext/zstd/CHANGELOG view on Meta::CPAN
cli : improved --list output
cli : new : can split input file for dictionary training, using command -B#
cli : new : clean operation artefact on Ctrl-C interruption
cli : fix : do not change /dev/null permissions when using command -t with root access, reported by @mike155 (#851)
cli : fix : write file size in header in multiple-files mode
api : added macro ZSTD_COMPRESSBOUND() for static allocation
api : experimental : new advanced decompression API
api : fix : sizeof_CCtx() used to over-estimate
build: fix : no-multithread variant compiles without pool.c dependency, reported by Mitchell Blank Jr (@mitchblank) (#819)
build: better compatibility with reproducible builds, by Bernhard M. Wiedemann (@bmwiedemann) (#818)
example : added streaming_memory_usage
license : changed /examples license to BSD + GPLv2
license : fix a few header files to reflect new license (#825)
v1.3.1
New license : BSD + GPLv2
perf: substantially decreased memory usage in Multi-threading mode, thanks to reports by Tino Reichardt (@mcmilk)
perf: Multi-threading supports up to 256 threads. Cap at 256 when more are requested (#760)
cli : improved and fixed --list command, by @ib (#772)
cli : command -vV to list supported formats, by @ib (#771)
build : fixed binary variants, reported by @svenha (#788)
ext/zstd/CHANGELOG view on Meta::CPAN
build: enabled Multi-threading support for *BSD, by Baptiste Daroussin
tools: updated Paramgrill. Command -O# provides best parameters for sample and speed target.
new : contrib/linux-kernel version, by Nick Terrell
v1.1.4
cli : new : can compress in *.gz format, using --format=gzip command, by Przemyslaw Skibinski
cli : new : advanced benchmark command --priority=rt
cli : fix : write on sparse-enabled file systems in 32-bits mode, by @ds77
cli : fix : --rm remains silent when input is stdin
cli : experimental : xzstd, with support for xz/lzma decoding, by Przemyslaw Skibinski
speed : improved decompression speed in streaming mode for single shot scenarios (+5%)
memory: DDict (decompression dictionary) memory usage down from 150 KB to 20 KB
arch: 32-bits variant able to generate and decode very long matches (>32 MB), by Sean Purcell
API : new : ZSTD_findFrameCompressedSize(), ZSTD_getFrameContentSize(), ZSTD_findDecompressedSize()
API : changed : dropped support of legacy versions <= v0.3 (can be changed by modifying ZSTD_LEGACY_SUPPORT value)
build : new: meson build system in contrib/meson, by Dima Krasner
build : improved cmake script, by @Majlen
build : added -Wformat-security flag, as recommended by Padraig Brady
doc : new : educational decoder, by Sean Purcell
v1.1.3
ext/zstd/CHANGELOG view on Meta::CPAN
dictBuilder : improved dictionary generation quality, thanks to Nick Terrell
API : new : lib/compress/ZSTDMT_compress.h multithreading API (experimental)
API : new : ZSTD_create?Dict_byReference(), requested by Bartosz Taudul
API : new : ZDICT_finalizeDictionary()
API : fix : ZSTD_initCStream_usingCDict() properly writes dictID into frame header, by Gregory Szorc (#511)
API : fix : all symbols properly exposed in libzstd, by Nick Terrell
build : support for Solaris target, by Przemyslaw Skibinski
doc : clarified specification, by Sean Purcell
v1.1.2
API : streaming : decompression : changed : automatic implicit reset when chain-decoding new frames without init
API : experimental : added : dictID retrieval functions, and ZSTD_initCStream_srcSize()
API : zbuff : changed : prototypes now generate deprecation warnings
lib : improved : faster decompression speed at ultra compression settings and 32-bits mode
lib : changed : only public ZSTD_ symbols are now exposed
lib : changed : reduced usage of stack memory
lib : fixed : several corner case bugs, by Nick Terrell
cli : new : gzstd, experimental version able to decode .gz files, by Przemyslaw Skibinski
cli : new : preserve file attributes
cli : new : added zstdless and zstdgrep tools
cli : fixed : status displays total amount decoded, even for file consisting of multiple frames (like pzstd)
cli : fixed : zstdcat
zlib_wrapper : added support for gz* functions, by Przemyslaw Skibinski
install : better compatibility with FreeBSD, by Dimitry Andric
source tree : changed : zbuff source files moved to lib/deprecated
v1.1.1
New : command -M#, --memory=, --memlimit=, --memlimit-decompress= to limit allowed memory consumption
New : doc/zstd_manual.html, by Przemyslaw Skibinski
Improved : slightly better compression ratio at --ultra levels (>= 20)
Improved : better memory usage when using streaming compression API, thanks to @Rogier-5 report
Added : API : ZSTD_initCStream_usingCDict(), ZSTD_initDStream_usingDDict() (experimental section)
Added : example/multiple_streaming_compression.c
Changed : zstd_errors.h is now installed within /include (and replaces errors_public.h)
Updated man page
Fixed : zstd-small, zstd-compress and zstd-decompress compilation targets
v1.1.0
New : contrib/pzstd, parallel version of zstd, by Nick Terrell
added : NetBSD install target (#338)
Improved : speed for batches of small files
Improved : speed of zlib wrapper, by Przemyslaw Skibinski
Changed : libzstd on Windows supports legacy formats, by Christophe Chevalier
ext/zstd/CHANGELOG view on Meta::CPAN
v1.0.0
Change Licensing, all project is now BSD, Copyright Facebook
Small decompression speed improvement
API : Streaming API supports legacy format
API : ZDICT_getDictID(), ZSTD_sizeof_{CCtx, DCtx, CStream, DStream}(), ZSTD_setDStreamParameter()
CLI supports legacy formats v0.4+
Fixed : compression fails on certain huge files, reported by Jesse McGrew
Enhanced documentation, by Przemyslaw Skibinski
v0.8.1
New streaming API
Changed : --ultra now enables levels beyond 19
Changed : -i# now selects benchmark time in second
Fixed : ZSTD_compress* can now compress > 4 GB in a single pass, reported by Nick Terrell
Fixed : speed regression on specific patterns (#272)
Fixed : support for Z_SYNC_FLUSH, by Dmitry Krot (#291)
Fixed : ICC compilation, by Przemyslaw Skibinski
v0.8.0
Improved : better speed on clang and gcc -O2, thanks to Eric Biggers
New : Build on FreeBSD and DragonFly, thanks to JrMarino
ext/zstd/CHANGELOG view on Meta::CPAN
v0.5.1
New : Optimal parsing => Very high compression modes, thanks to Przemyslaw Skibinski
Changed : Dictionary builder integrated into libzstd and zstd cli
Changed (!) : zstd cli now uses "multiple input files" as default mode. See `zstd -h`.
Fix : high compression modes for big-endian platforms
New : zstd cli : `-t` | `--test` command
v0.5.0
New : dictionary builder utility
Changed : streaming & dictionary API
Improved : better compression of small data
v0.4.7
Improved : small compression speed improvement in HC mode
Changed : `zstd_decompress.c` has ZSTD_LEGACY_SUPPORT to 0 by default
fix : bt search bug
v0.4.6
fix : fast compression mode on Windows
New : cmake configuration file, thanks to Artyom Dymchenko
ext/zstd/TESTING.md view on Meta::CPAN
- Small tests (`tests/legacy.c`, `tests/longmatch.c`, `tests/symbols.c`) on x64_64
Medium Tests
------------
Medium tests run on every commit and pull request to `dev` branch, on TravisCI.
They consist of the following tests:
- The following tests run with UBsan and Asan on x86_64 and x86, as well as with
Msan on x86_64
- `tests/playTests.sh --test-long-data`
- Fuzzer tests: `tests/fuzzer.c`, `tests/zstreamtest.c`, and `tests/decodecorpus.c`
- `tests/zstreamtest.c` under Tsan (streaming mode, including multithreaded mode)
- Valgrind Test (`make -C tests valgrindTest`) (testing CLI and fuzzer under valgrind)
- Fuzzer tests (see above) on ARM, AArch64, PowerPC, and PowerPC64
Long Tests
----------
Long tests run on all commits to `master` branch,
and once a day on the current version of `dev` branch,
on TravisCI.
They consist of the following tests:
- Entire test suite (including fuzzers and some other specialized tests) on:
- x86_64 and x86 with UBsan and Asan
- x86_64 with Msan
- ARM, AArch64, PowerPC, and PowerPC64
- Streaming mode fuzzer with Tsan (for the `zstdmt` testing)
- ZlibWrapper tests, including under valgrind
- Versions test (ensuring `zstd` can decode files from all previous versions)
- `pzstd` with asan and tsan, as well as in 32-bits mode
- Testing `zstd` with legacy mode off
- Testing `zbuff` (old streaming API)
- Entire test suite and make install on macOS
ext/zstd/contrib/adaptive-compression/test-correctness.sh view on Meta::CPAN
zstd -d tmp.zst -o tmp2
diff -s -q tmp tmp2
rm tmp*
./datagen -s9 -g500KB > tmp
./adapt -otmp.zst tmp
zstd -d tmp.zst -o tmp2
diff -s -q tmp tmp2
rm tmp*
echo -e "\ncorrectness tests -- streaming"
./datagen -s10 -g1GB > tmp
cat tmp | ./adapt > tmp.zst
zstd -d tmp.zst -o tmp2
diff -s -q tmp tmp2
rm tmp*
./datagen -s11 -g100MB > tmp
cat tmp | ./adapt > tmp.zst
zstd -d tmp.zst -o tmp2
diff -s -q tmp tmp2
ext/zstd/contrib/linux-kernel/0002-lib-Add-zstd-modules.patch view on Meta::CPAN
+ * Return: A decompression context emplaced into workspace.
+ */
+ZSTD_DCtx *ZSTD_initDCtx(void *workspace, size_t workspaceSize);
+
+/**
+ * ZSTD_decompressDCtx() - decompress zstd compressed src into dst
+ * @ctx: The decompression context.
+ * @dst: The buffer to decompress src into.
+ * @dstCapacity: The size of the destination buffer. Must be at least as large
+ * as the decompressed size. If the caller cannot upper bound the
+ * decompressed size, then it's better to use the streaming API.
+ * @src: The zstd compressed data to decompress. Multiple concatenated
+ * frames and skippable frames are allowed.
+ * @srcSize: The exact size of the data to decompress.
+ *
+ * Return: The decompressed size or an error, which can be checked using
+ * ZSTD_isError().
+ */
+size_t ZSTD_decompressDCtx(ZSTD_DCtx *ctx, void *dst, size_t dstCapacity,
+ const void *src, size_t srcSize);
+
ext/zstd/contrib/linux-kernel/0002-lib-Add-zstd-modules.patch view on Meta::CPAN
+size_t ZSTD_compress_usingDict(ZSTD_CCtx *ctx, void *dst, size_t dstCapacity,
+ const void *src, size_t srcSize, const void *dict, size_t dictSize,
+ ZSTD_parameters params);
+
+/**
+ * ZSTD_decompress_usingDict() - decompress src into dst using a dictionary
+ * @ctx: The decompression context.
+ * @dst: The buffer to decompress src into.
+ * @dstCapacity: The size of the destination buffer. Must be at least as large
+ * as the decompressed size. If the caller cannot upper bound the
+ * decompressed size, then it's better to use the streaming API.
+ * @src: The zstd compressed data to decompress. Multiple concatenated
+ * frames and skippable frames are allowed.
+ * @srcSize: The exact size of the data to decompress.
+ * @dict: The dictionary to use for decompression. The same dictionary
+ * must've been used to compress the data.
+ * @dictSize: The size of the dictionary.
+ *
+ * Return: The decompressed size or an error, which can be checked using
+ * ZSTD_isError().
+ */
ext/zstd/contrib/linux-kernel/0002-lib-Add-zstd-modules.patch view on Meta::CPAN
+ */
+ZSTD_DDict *ZSTD_initDDict(const void *dictBuffer, size_t dictSize,
+ void *workspace, size_t workspaceSize);
+
+/**
+ * ZSTD_decompress_usingDDict() - decompress src into dst using a ZSTD_DDict
+ * @ctx: The decompression context.
+ * @dst: The buffer to decompress src into.
+ * @dstCapacity: The size of the destination buffer. Must be at least as large
+ * as the decompressed size. If the caller cannot upper bound the
+ * decompressed size, then it's better to use the streaming API.
+ * @src: The zstd compressed data to decompress. Multiple concatenated
+ * frames and skippable frames are allowed.
+ * @srcSize: The exact size of the data to decompress.
+ * @ddict: The digested dictionary to use for decompression. The same
+ * dictionary must've been used to compress the data.
+ *
+ * Return: The decompressed size or an error, which can be checked using
+ * ZSTD_isError().
+ */
+size_t ZSTD_decompress_usingDDict(ZSTD_DCtx *dctx, void *dst,
+ size_t dstCapacity, const void *src, size_t srcSize,
+ const ZSTD_DDict *ddict);
+
+
+/*-**************************
+ * Streaming
+ ***************************/
+
+/**
+ * struct ZSTD_inBuffer - input buffer for streaming
+ * @src: Start of the input buffer.
+ * @size: Size of the input buffer.
+ * @pos: Position where reading stopped. Will be updated.
+ * Necessarily 0 <= pos <= size.
+ */
+typedef struct ZSTD_inBuffer_s {
+ const void *src;
+ size_t size;
+ size_t pos;
+} ZSTD_inBuffer;
+
+/**
+ * struct ZSTD_outBuffer - output buffer for streaming
+ * @dst: Start of the output buffer.
+ * @size: Size of the output buffer.
+ * @pos: Position where writing stopped. Will be updated.
+ * Necessarily 0 <= pos <= size.
+ */
+typedef struct ZSTD_outBuffer_s {
+ void *dst;
+ size_t size;
+ size_t pos;
+} ZSTD_outBuffer;
+
+
+
+/*-*****************************************************************************
+ * Streaming compression - HowTo
+ *
+ * A ZSTD_CStream object is required to track streaming operation.
+ * Use ZSTD_initCStream() to initialize a ZSTD_CStream object.
+ * ZSTD_CStream objects can be reused multiple times on consecutive compression
+ * operations. It is recommended to re-use ZSTD_CStream in situations where many
+ * streaming operations will be achieved consecutively. Use one separate
+ * ZSTD_CStream per thread for parallel execution.
+ *
+ * Use ZSTD_compressStream() repetitively to consume input stream.
+ * The function will automatically update both `pos` fields.
+ * Note that it may not consume the entire input, in which case `pos < size`,
+ * and it's up to the caller to present again remaining data.
+ * It returns a hint for the preferred number of bytes to use as an input for
+ * the next function call.
+ *
+ * At any moment, it's possible to flush whatever data remains within internal
ext/zstd/contrib/linux-kernel/0002-lib-Add-zstd-modules.patch view on Meta::CPAN
+/**
+ * ZSTD_CStreamWorkspaceBound() - memory needed to initialize a ZSTD_CStream
+ * @cParams: The compression parameters to be used for compression.
+ *
+ * Return: A lower bound on the size of the workspace that is passed to
+ * ZSTD_initCStream() and ZSTD_initCStream_usingCDict().
+ */
+size_t ZSTD_CStreamWorkspaceBound(ZSTD_compressionParameters cParams);
+
+/**
+ * struct ZSTD_CStream - the zstd streaming compression context
+ */
+typedef struct ZSTD_CStream_s ZSTD_CStream;
+
+/*===== ZSTD_CStream management functions =====*/
+/**
+ * ZSTD_initCStream() - initialize a zstd streaming compression context
+ * @params: The zstd compression parameters.
+ * @pledgedSrcSize: If params.fParams.contentSizeFlag == 1 then the caller must
+ * pass the source size (zero means empty source). Otherwise,
+ * the caller may optionally pass the source size, or zero if
+ * unknown.
+ * @workspace: The workspace to emplace the context into. It must outlive
+ * the returned context.
+ * @workspaceSize: The size of workspace.
+ * Use ZSTD_CStreamWorkspaceBound(params.cParams) to determine
+ * how large the workspace must be.
+ *
+ * Return: The zstd streaming compression context.
+ */
+ZSTD_CStream *ZSTD_initCStream(ZSTD_parameters params,
+ unsigned long long pledgedSrcSize, void *workspace,
+ size_t workspaceSize);
+
+/**
+ * ZSTD_initCStream_usingCDict() - initialize a streaming compression context
+ * @cdict: The digested dictionary to use for compression.
+ * @pledgedSrcSize: Optionally the source size, or zero if unknown.
+ * @workspace: The workspace to emplace the context into. It must outlive
+ * the returned context.
+ * @workspaceSize: The size of workspace. Call ZSTD_CStreamWorkspaceBound()
+ * with the cParams used to initialize the cdict to determine
+ * how large the workspace must be.
+ *
+ * Return: The zstd streaming compression context.
+ */
+ZSTD_CStream *ZSTD_initCStream_usingCDict(const ZSTD_CDict *cdict,
+ unsigned long long pledgedSrcSize, void *workspace,
+ size_t workspaceSize);
+
+/*===== Streaming compression functions =====*/
+/**
+ * ZSTD_resetCStream() - reset the context using parameters from creation
+ * @zcs: The zstd streaming compression context to reset.
+ * @pledgedSrcSize: Optionally the source size, or zero if unknown.
+ *
+ * Resets the context using the parameters from creation. Skips dictionary
+ * loading, since it can be reused. If `pledgedSrcSize` is non-zero the frame
+ * content size is always written into the frame header.
+ *
+ * Return: Zero or an error, which can be checked using ZSTD_isError().
+ */
+size_t ZSTD_resetCStream(ZSTD_CStream *zcs, unsigned long long pledgedSrcSize);
+/**
+ * ZSTD_compressStream() - streaming compress some of input into output
+ * @zcs: The zstd streaming compression context.
+ * @output: Destination buffer. `output->pos` is updated to indicate how much
+ * compressed data was written.
+ * @input: Source buffer. `input->pos` is updated to indicate how much data was
+ * read. Note that it may not consume the entire input, in which case
+ * `input->pos < input->size`, and it's up to the caller to present
+ * remaining data again.
+ *
+ * The `input` and `output` buffers may be any size. Guaranteed to make some
+ * forward progress if `input` and `output` are not empty.
+ *
+ * Return: A hint for the number of bytes to use as the input for the next
+ * function call or an error, which can be checked using
+ * ZSTD_isError().
+ */
+size_t ZSTD_compressStream(ZSTD_CStream *zcs, ZSTD_outBuffer *output,
+ ZSTD_inBuffer *input);
+/**
+ * ZSTD_flushStream() - flush internal buffers into output
+ * @zcs: The zstd streaming compression context.
+ * @output: Destination buffer. `output->pos` is updated to indicate how much
+ * compressed data was written.
+ *
+ * ZSTD_flushStream() must be called until it returns 0, meaning all the data
+ * has been flushed. Since ZSTD_flushStream() causes a block to be ended,
+ * calling it too often will degrade the compression ratio.
+ *
+ * Return: The number of bytes still present within internal buffers or an
+ * error, which can be checked using ZSTD_isError().
+ */
+size_t ZSTD_flushStream(ZSTD_CStream *zcs, ZSTD_outBuffer *output);
+/**
+ * ZSTD_endStream() - flush internal buffers into output and end the frame
+ * @zcs: The zstd streaming compression context.
+ * @output: Destination buffer. `output->pos` is updated to indicate how much
+ * compressed data was written.
+ *
+ * ZSTD_endStream() must be called until it returns 0, meaning all the data has
+ * been flushed and the frame epilogue has been written.
+ *
+ * Return: The number of bytes still present within internal buffers or an
+ * error, which can be checked using ZSTD_isError().
+ */
+size_t ZSTD_endStream(ZSTD_CStream *zcs, ZSTD_outBuffer *output);
ext/zstd/contrib/linux-kernel/0002-lib-Add-zstd-modules.patch view on Meta::CPAN
+ *
+ * Return: The recommended size for the output buffer.
+ */
+size_t ZSTD_CStreamOutSize(void);
+
+
+
+/*-*****************************************************************************
+ * Streaming decompression - HowTo
+ *
+ * A ZSTD_DStream object is required to track streaming operations.
+ * Use ZSTD_initDStream() to initialize a ZSTD_DStream object.
+ * ZSTD_DStream objects can be re-used multiple times.
+ *
+ * Use ZSTD_decompressStream() repetitively to consume your input.
+ * The function will update both `pos` fields.
+ * If `input->pos < input->size`, some input has not been consumed.
+ * It's up to the caller to present again remaining data.
+ * If `output->pos < output->size`, decoder has flushed everything it could.
+ * Returns 0 iff a frame is completely decoded and fully flushed.
+ * Otherwise it returns a suggested next input size that will never load more
ext/zstd/contrib/linux-kernel/0002-lib-Add-zstd-modules.patch view on Meta::CPAN
+/**
+ * ZSTD_DStreamWorkspaceBound() - memory needed to initialize a ZSTD_DStream
+ * @maxWindowSize: The maximum window size allowed for compressed frames.
+ *
+ * Return: A lower bound on the size of the workspace that is passed to
+ * ZSTD_initDStream() and ZSTD_initDStream_usingDDict().
+ */
+size_t ZSTD_DStreamWorkspaceBound(size_t maxWindowSize);
+
+/**
+ * struct ZSTD_DStream - the zstd streaming decompression context
+ */
+typedef struct ZSTD_DStream_s ZSTD_DStream;
+/*===== ZSTD_DStream management functions =====*/
+/**
+ * ZSTD_initDStream() - initialize a zstd streaming decompression context
+ * @maxWindowSize: The maximum window size allowed for compressed frames.
+ * @workspace: The workspace to emplace the context into. It must outlive
+ * the returned context.
+ * @workspaceSize: The size of workspace.
+ * Use ZSTD_DStreamWorkspaceBound(maxWindowSize) to determine
+ * how large the workspace must be.
+ *
+ * Return: The zstd streaming decompression context.
+ */
+ZSTD_DStream *ZSTD_initDStream(size_t maxWindowSize, void *workspace,
+ size_t workspaceSize);
+/**
+ * ZSTD_initDStream_usingDDict() - initialize streaming decompression context
+ * @maxWindowSize: The maximum window size allowed for compressed frames.
+ * @ddict: The digested dictionary to use for decompression.
+ * @workspace: The workspace to emplace the context into. It must outlive
+ * the returned context.
+ * @workspaceSize: The size of workspace.
+ * Use ZSTD_DStreamWorkspaceBound(maxWindowSize) to determine
+ * how large the workspace must be.
+ *
+ * Return: The zstd streaming decompression context.
+ */
+ZSTD_DStream *ZSTD_initDStream_usingDDict(size_t maxWindowSize,
+ const ZSTD_DDict *ddict, void *workspace, size_t workspaceSize);
+
+/*===== Streaming decompression functions =====*/
+/**
+ * ZSTD_resetDStream() - reset the context using parameters from creation
+ * @zds: The zstd streaming decompression context to reset.
+ *
+ * Resets the context using the parameters from creation. Skips dictionary
+ * loading, since it can be reused.
+ *
+ * Return: Zero or an error, which can be checked using ZSTD_isError().
+ */
+size_t ZSTD_resetDStream(ZSTD_DStream *zds);
+/**
+ * ZSTD_decompressStream() - streaming decompress some of input into output
+ * @zds: The zstd streaming decompression context.
+ * @output: Destination buffer. `output.pos` is updated to indicate how much
+ * decompressed data was written.
+ * @input: Source buffer. `input.pos` is updated to indicate how much data was
+ * read. Note that it may not consume the entire input, in which case
+ * `input.pos < input.size`, and it's up to the caller to present
+ * remaining data again.
+ *
+ * The `input` and `output` buffers may be any size. Guaranteed to make some
+ * forward progress if `input` and `output` are not empty.
+ * ZSTD_decompressStream() will not consume the last byte of the frame until
ext/zstd/contrib/linux-kernel/0002-lib-Add-zstd-modules.patch view on Meta::CPAN
+ * always large enough to succeed.
+ *
+ * Return: 0 on success. If more data is required it returns how many bytes
+ * must be provided to make forward progress. Otherwise it returns
+ * an error, which can be checked using ZSTD_isError().
+ */
+size_t ZSTD_getFrameParams(ZSTD_frameParams *fparamsPtr, const void *src,
+ size_t srcSize);
+
+/*-*****************************************************************************
+ * Buffer-less and synchronous inner streaming functions
+ *
+ * This is an advanced API, giving full control over buffer management, for
+ * users which need direct control over memory.
+ * But it's also a complex one, with many restrictions (documented below).
+ * Prefer using normal streaming API for an easier experience
+ ******************************************************************************/
+
+/*-*****************************************************************************
+ * Buffer-less streaming compression (synchronous mode)
+ *
+ * A ZSTD_CCtx object is required to track streaming operations.
+ * Use ZSTD_initCCtx() to initialize a context.
+ * ZSTD_CCtx object can be re-used multiple times within successive compression
+ * operations.
+ *
+ * Start by initializing a context.
+ * Use ZSTD_compressBegin(), or ZSTD_compressBegin_usingDict() for dictionary
+ * compression,
+ * or ZSTD_compressBegin_advanced(), for finer parameter control.
+ * It's also possible to duplicate a reference context which has already been
+ * initialized, using ZSTD_copyCCtx()
ext/zstd/contrib/linux-kernel/0002-lib-Add-zstd-modules.patch view on Meta::CPAN
+ *
+ * Finish a frame with ZSTD_compressEnd(), which will write the last block(s)
+ * and optional checksum. It's possible to use srcSize==0, in which case, it
+ * will write a final empty block to end the frame. Without last block mark,
+ * frames will be considered unfinished (corrupted) by decoders.
+ *
+ * `ZSTD_CCtx` object can be re-used (ZSTD_compressBegin()) to compress some new
+ * frame.
+ ******************************************************************************/
+
+/*===== Buffer-less streaming compression functions =====*/
+size_t ZSTD_compressBegin(ZSTD_CCtx *cctx, int compressionLevel);
+size_t ZSTD_compressBegin_usingDict(ZSTD_CCtx *cctx, const void *dict,
+ size_t dictSize, int compressionLevel);
+size_t ZSTD_compressBegin_advanced(ZSTD_CCtx *cctx, const void *dict,
+ size_t dictSize, ZSTD_parameters params,
+ unsigned long long pledgedSrcSize);
+size_t ZSTD_copyCCtx(ZSTD_CCtx *cctx, const ZSTD_CCtx *preparedCCtx,
+ unsigned long long pledgedSrcSize);
+size_t ZSTD_compressBegin_usingCDict(ZSTD_CCtx *cctx, const ZSTD_CDict *cdict,
+ unsigned long long pledgedSrcSize);
+size_t ZSTD_compressContinue(ZSTD_CCtx *cctx, void *dst, size_t dstCapacity,
+ const void *src, size_t srcSize);
+size_t ZSTD_compressEnd(ZSTD_CCtx *cctx, void *dst, size_t dstCapacity,
+ const void *src, size_t srcSize);
+
+
+
+/*-*****************************************************************************
+ * Buffer-less streaming decompression (synchronous mode)
+ *
+ * A ZSTD_DCtx object is required to track streaming operations.
+ * Use ZSTD_initDCtx() to initialize a context.
+ * A ZSTD_DCtx object can be re-used multiple times.
+ *
+ * First typical operation is to retrieve frame parameters, using
+ * ZSTD_getFrameParams(). It fills a ZSTD_frameParams structure which provide
+ * important information to correctly decode the frame, such as the minimum
+ * rolling buffer size to allocate to decompress data (`windowSize`), and the
+ * dictionary ID used.
+ * Note: content size is optional, it may not be present. 0 means unknown.
+ * Note that these values could be wrong, either because of data malformation,
ext/zstd/contrib/linux-kernel/0002-lib-Add-zstd-modules.patch view on Meta::CPAN
+ * For skippable frames ZSTD_decompressContinue() always returns 0.
+ * For skippable frames ZSTD_getFrameParams() returns fparamsPtr->windowLog==0
+ * what means that a frame is skippable.
+ * Note: If fparamsPtr->frameContentSize==0, it is ambiguous: the frame might
+ * actually be a zstd encoded frame with no content. For purposes of
+ * decompression, it is valid in both cases to skip the frame using
+ * ZSTD_findFrameCompressedSize() to find its size in bytes.
+ * It also returns frame size as fparamsPtr->frameContentSize.
+ ******************************************************************************/
+
+/*===== Buffer-less streaming decompression functions =====*/
+size_t ZSTD_decompressBegin(ZSTD_DCtx *dctx);
+size_t ZSTD_decompressBegin_usingDict(ZSTD_DCtx *dctx, const void *dict,
+ size_t dictSize);
+void ZSTD_copyDCtx(ZSTD_DCtx *dctx, const ZSTD_DCtx *preparedDCtx);
+size_t ZSTD_nextSrcSizeToDecompress(ZSTD_DCtx *dctx);
+size_t ZSTD_decompressContinue(ZSTD_DCtx *dctx, void *dst, size_t dstCapacity,
+ const void *src, size_t srcSize);
+typedef enum {
+ ZSTDnit_frameHeader,
+ ZSTDnit_blockHeader,
ext/zstd/contrib/linux-kernel/include/linux/zstd.h view on Meta::CPAN
* Return: A decompression context emplaced into workspace.
*/
ZSTD_DCtx *ZSTD_initDCtx(void *workspace, size_t workspaceSize);
/**
* ZSTD_decompressDCtx() - decompress zstd compressed src into dst
* @ctx: The decompression context.
* @dst: The buffer to decompress src into.
* @dstCapacity: The size of the destination buffer. Must be at least as large
* as the decompressed size. If the caller cannot upper bound the
* decompressed size, then it's better to use the streaming API.
* @src: The zstd compressed data to decompress. Multiple concatenated
* frames and skippable frames are allowed.
* @srcSize: The exact size of the data to decompress.
*
* Return: The decompressed size or an error, which can be checked using
* ZSTD_isError().
*/
size_t ZSTD_decompressDCtx(ZSTD_DCtx *ctx, void *dst, size_t dstCapacity,
const void *src, size_t srcSize);
ext/zstd/contrib/linux-kernel/include/linux/zstd.h view on Meta::CPAN
size_t ZSTD_compress_usingDict(ZSTD_CCtx *ctx, void *dst, size_t dstCapacity,
const void *src, size_t srcSize, const void *dict, size_t dictSize,
ZSTD_parameters params);
/**
* ZSTD_decompress_usingDict() - decompress src into dst using a dictionary
* @ctx: The decompression context.
* @dst: The buffer to decompress src into.
* @dstCapacity: The size of the destination buffer. Must be at least as large
* as the decompressed size. If the caller cannot upper bound the
* decompressed size, then it's better to use the streaming API.
* @src: The zstd compressed data to decompress. Multiple concatenated
* frames and skippable frames are allowed.
* @srcSize: The exact size of the data to decompress.
* @dict: The dictionary to use for decompression. The same dictionary
* must've been used to compress the data.
* @dictSize: The size of the dictionary.
*
* Return: The decompressed size or an error, which can be checked using
* ZSTD_isError().
*/
ext/zstd/contrib/linux-kernel/include/linux/zstd.h view on Meta::CPAN
*/
ZSTD_DDict *ZSTD_initDDict(const void *dictBuffer, size_t dictSize,
void *workspace, size_t workspaceSize);
/**
* ZSTD_decompress_usingDDict() - decompress src into dst using a ZSTD_DDict
* @ctx: The decompression context.
* @dst: The buffer to decompress src into.
* @dstCapacity: The size of the destination buffer. Must be at least as large
* as the decompressed size. If the caller cannot upper bound the
* decompressed size, then it's better to use the streaming API.
* @src: The zstd compressed data to decompress. Multiple concatenated
* frames and skippable frames are allowed.
* @srcSize: The exact size of the data to decompress.
* @ddict: The digested dictionary to use for decompression. The same
* dictionary must've been used to compress the data.
*
* Return: The decompressed size or an error, which can be checked using
* ZSTD_isError().
*/
size_t ZSTD_decompress_usingDDict(ZSTD_DCtx *dctx, void *dst,
size_t dstCapacity, const void *src, size_t srcSize,
const ZSTD_DDict *ddict);
/*-**************************
* Streaming
***************************/
/**
* struct ZSTD_inBuffer - input buffer for streaming
* @src: Start of the input buffer.
* @size: Size of the input buffer.
* @pos: Position where reading stopped. Will be updated.
* Necessarily 0 <= pos <= size.
*/
typedef struct ZSTD_inBuffer_s {
const void *src;
size_t size;
size_t pos;
} ZSTD_inBuffer;
/**
* struct ZSTD_outBuffer - output buffer for streaming
* @dst: Start of the output buffer.
* @size: Size of the output buffer.
* @pos: Position where writing stopped. Will be updated.
* Necessarily 0 <= pos <= size.
*/
typedef struct ZSTD_outBuffer_s {
void *dst;
size_t size;
size_t pos;
} ZSTD_outBuffer;
/*-*****************************************************************************
* Streaming compression - HowTo
*
* A ZSTD_CStream object is required to track streaming operation.
* Use ZSTD_initCStream() to initialize a ZSTD_CStream object.
* ZSTD_CStream objects can be reused multiple times on consecutive compression
* operations. It is recommended to re-use ZSTD_CStream in situations where many
* streaming operations will be achieved consecutively. Use one separate
* ZSTD_CStream per thread for parallel execution.
*
* Use ZSTD_compressStream() repetitively to consume input stream.
* The function will automatically update both `pos` fields.
* Note that it may not consume the entire input, in which case `pos < size`,
* and it's up to the caller to present again remaining data.
* It returns a hint for the preferred number of bytes to use as an input for
* the next function call.
*
* At any moment, it's possible to flush whatever data remains within internal
ext/zstd/contrib/linux-kernel/include/linux/zstd.h view on Meta::CPAN
/**
* ZSTD_CStreamWorkspaceBound() - memory needed to initialize a ZSTD_CStream
* @cParams: The compression parameters to be used for compression.
*
* Return: A lower bound on the size of the workspace that is passed to
* ZSTD_initCStream() and ZSTD_initCStream_usingCDict().
*/
size_t ZSTD_CStreamWorkspaceBound(ZSTD_compressionParameters cParams);
/**
* struct ZSTD_CStream - the zstd streaming compression context
*/
typedef struct ZSTD_CStream_s ZSTD_CStream;
/*===== ZSTD_CStream management functions =====*/
/**
* ZSTD_initCStream() - initialize a zstd streaming compression context
* @params: The zstd compression parameters.
* @pledgedSrcSize: If params.fParams.contentSizeFlag == 1 then the caller must
* pass the source size (zero means empty source). Otherwise,
* the caller may optionally pass the source size, or zero if
* unknown.
* @workspace: The workspace to emplace the context into. It must outlive
* the returned context.
* @workspaceSize: The size of workspace.
* Use ZSTD_CStreamWorkspaceBound(params.cParams) to determine
* how large the workspace must be.
*
* Return: The zstd streaming compression context.
*/
ZSTD_CStream *ZSTD_initCStream(ZSTD_parameters params,
unsigned long long pledgedSrcSize, void *workspace,
size_t workspaceSize);
/**
* ZSTD_initCStream_usingCDict() - initialize a streaming compression context
* @cdict: The digested dictionary to use for compression.
* @pledgedSrcSize: Optionally the source size, or zero if unknown.
* @workspace: The workspace to emplace the context into. It must outlive
* the returned context.
* @workspaceSize: The size of workspace. Call ZSTD_CStreamWorkspaceBound()
* with the cParams used to initialize the cdict to determine
* how large the workspace must be.
*
* Return: The zstd streaming compression context.
*/
ZSTD_CStream *ZSTD_initCStream_usingCDict(const ZSTD_CDict *cdict,
unsigned long long pledgedSrcSize, void *workspace,
size_t workspaceSize);
/*===== Streaming compression functions =====*/
/**
* ZSTD_resetCStream() - reset the context using parameters from creation
* @zcs: The zstd streaming compression context to reset.
* @pledgedSrcSize: Optionally the source size, or zero if unknown.
*
* Resets the context using the parameters from creation. Skips dictionary
* loading, since it can be reused. If `pledgedSrcSize` is non-zero the frame
* content size is always written into the frame header.
*
* Return: Zero or an error, which can be checked using ZSTD_isError().
*/
size_t ZSTD_resetCStream(ZSTD_CStream *zcs, unsigned long long pledgedSrcSize);
/**
* ZSTD_compressStream() - streaming compress some of input into output
* @zcs: The zstd streaming compression context.
* @output: Destination buffer. `output->pos` is updated to indicate how much
* compressed data was written.
* @input: Source buffer. `input->pos` is updated to indicate how much data was
* read. Note that it may not consume the entire input, in which case
* `input->pos < input->size`, and it's up to the caller to present
* remaining data again.
*
* The `input` and `output` buffers may be any size. Guaranteed to make some
* forward progress if `input` and `output` are not empty.
*
* Return: A hint for the number of bytes to use as the input for the next
* function call or an error, which can be checked using
* ZSTD_isError().
*/
size_t ZSTD_compressStream(ZSTD_CStream *zcs, ZSTD_outBuffer *output,
ZSTD_inBuffer *input);
/**
* ZSTD_flushStream() - flush internal buffers into output
* @zcs: The zstd streaming compression context.
* @output: Destination buffer. `output->pos` is updated to indicate how much
* compressed data was written.
*
* ZSTD_flushStream() must be called until it returns 0, meaning all the data
* has been flushed. Since ZSTD_flushStream() causes a block to be ended,
* calling it too often will degrade the compression ratio.
*
* Return: The number of bytes still present within internal buffers or an
* error, which can be checked using ZSTD_isError().
*/
size_t ZSTD_flushStream(ZSTD_CStream *zcs, ZSTD_outBuffer *output);
/**
* ZSTD_endStream() - flush internal buffers into output and end the frame
* @zcs: The zstd streaming compression context.
* @output: Destination buffer. `output->pos` is updated to indicate how much
* compressed data was written.
*
* ZSTD_endStream() must be called until it returns 0, meaning all the data has
* been flushed and the frame epilogue has been written.
*
* Return: The number of bytes still present within internal buffers or an
* error, which can be checked using ZSTD_isError().
*/
size_t ZSTD_endStream(ZSTD_CStream *zcs, ZSTD_outBuffer *output);
ext/zstd/contrib/linux-kernel/include/linux/zstd.h view on Meta::CPAN
*
* Return: The recommended size for the output buffer.
*/
size_t ZSTD_CStreamOutSize(void);
/*-*****************************************************************************
* Streaming decompression - HowTo
*
* A ZSTD_DStream object is required to track streaming operations.
* Use ZSTD_initDStream() to initialize a ZSTD_DStream object.
* ZSTD_DStream objects can be re-used multiple times.
*
* Use ZSTD_decompressStream() repetitively to consume your input.
* The function will update both `pos` fields.
* If `input->pos < input->size`, some input has not been consumed.
* It's up to the caller to present again remaining data.
* If `output->pos < output->size`, decoder has flushed everything it could.
* Returns 0 iff a frame is completely decoded and fully flushed.
* Otherwise it returns a suggested next input size that will never load more
ext/zstd/contrib/linux-kernel/include/linux/zstd.h view on Meta::CPAN
/**
* ZSTD_DStreamWorkspaceBound() - memory needed to initialize a ZSTD_DStream
* @maxWindowSize: The maximum window size allowed for compressed frames.
*
* Return: A lower bound on the size of the workspace that is passed to
* ZSTD_initDStream() and ZSTD_initDStream_usingDDict().
*/
size_t ZSTD_DStreamWorkspaceBound(size_t maxWindowSize);
/**
* struct ZSTD_DStream - the zstd streaming decompression context
*/
typedef struct ZSTD_DStream_s ZSTD_DStream;
/*===== ZSTD_DStream management functions =====*/
/**
* ZSTD_initDStream() - initialize a zstd streaming decompression context
* @maxWindowSize: The maximum window size allowed for compressed frames.
* @workspace: The workspace to emplace the context into. It must outlive
* the returned context.
* @workspaceSize: The size of workspace.
* Use ZSTD_DStreamWorkspaceBound(maxWindowSize) to determine
* how large the workspace must be.
*
* Return: The zstd streaming decompression context.
*/
ZSTD_DStream *ZSTD_initDStream(size_t maxWindowSize, void *workspace,
size_t workspaceSize);
/**
* ZSTD_initDStream_usingDDict() - initialize streaming decompression context
* @maxWindowSize: The maximum window size allowed for compressed frames.
* @ddict: The digested dictionary to use for decompression.
* @workspace: The workspace to emplace the context into. It must outlive
* the returned context.
* @workspaceSize: The size of workspace.
* Use ZSTD_DStreamWorkspaceBound(maxWindowSize) to determine
* how large the workspace must be.
*
* Return: The zstd streaming decompression context.
*/
ZSTD_DStream *ZSTD_initDStream_usingDDict(size_t maxWindowSize,
const ZSTD_DDict *ddict, void *workspace, size_t workspaceSize);
/*===== Streaming decompression functions =====*/
/**
* ZSTD_resetDStream() - reset the context using parameters from creation
* @zds: The zstd streaming decompression context to reset.
*
* Resets the context using the parameters from creation. Skips dictionary
* loading, since it can be reused.
*
* Return: Zero or an error, which can be checked using ZSTD_isError().
*/
size_t ZSTD_resetDStream(ZSTD_DStream *zds);
/**
* ZSTD_decompressStream() - streaming decompress some of input into output
* @zds: The zstd streaming decompression context.
* @output: Destination buffer. `output.pos` is updated to indicate how much
* decompressed data was written.
* @input: Source buffer. `input.pos` is updated to indicate how much data was
* read. Note that it may not consume the entire input, in which case
* `input.pos < input.size`, and it's up to the caller to present
* remaining data again.
*
* The `input` and `output` buffers may be any size. Guaranteed to make some
* forward progress if `input` and `output` are not empty.
* ZSTD_decompressStream() will not consume the last byte of the frame until
ext/zstd/contrib/linux-kernel/include/linux/zstd.h view on Meta::CPAN
* always large enough to succeed.
*
* Return: 0 on success. If more data is required it returns how many bytes
* must be provided to make forward progress. Otherwise it returns
* an error, which can be checked using ZSTD_isError().
*/
size_t ZSTD_getFrameParams(ZSTD_frameParams *fparamsPtr, const void *src,
size_t srcSize);
/*-*****************************************************************************
* Buffer-less and synchronous inner streaming functions
*
* This is an advanced API, giving full control over buffer management, for
* users which need direct control over memory.
* But it's also a complex one, with many restrictions (documented below).
* Prefer using normal streaming API for an easier experience
******************************************************************************/
/*-*****************************************************************************
* Buffer-less streaming compression (synchronous mode)
*
* A ZSTD_CCtx object is required to track streaming operations.
* Use ZSTD_initCCtx() to initialize a context.
* ZSTD_CCtx object can be re-used multiple times within successive compression
* operations.
*
* Start by initializing a context.
* Use ZSTD_compressBegin(), or ZSTD_compressBegin_usingDict() for dictionary
* compression,
* or ZSTD_compressBegin_advanced(), for finer parameter control.
* It's also possible to duplicate a reference context which has already been
* initialized, using ZSTD_copyCCtx()
ext/zstd/contrib/linux-kernel/include/linux/zstd.h view on Meta::CPAN
*
* Finish a frame with ZSTD_compressEnd(), which will write the last block(s)
* and optional checksum. It's possible to use srcSize==0, in which case, it
* will write a final empty block to end the frame. Without last block mark,
* frames will be considered unfinished (corrupted) by decoders.
*
* `ZSTD_CCtx` object can be re-used (ZSTD_compressBegin()) to compress some new
* frame.
******************************************************************************/
/*===== Buffer-less streaming compression functions =====*/
size_t ZSTD_compressBegin(ZSTD_CCtx *cctx, int compressionLevel);
size_t ZSTD_compressBegin_usingDict(ZSTD_CCtx *cctx, const void *dict,
size_t dictSize, int compressionLevel);
size_t ZSTD_compressBegin_advanced(ZSTD_CCtx *cctx, const void *dict,
size_t dictSize, ZSTD_parameters params,
unsigned long long pledgedSrcSize);
size_t ZSTD_copyCCtx(ZSTD_CCtx *cctx, const ZSTD_CCtx *preparedCCtx,
unsigned long long pledgedSrcSize);
size_t ZSTD_compressBegin_usingCDict(ZSTD_CCtx *cctx, const ZSTD_CDict *cdict,
unsigned long long pledgedSrcSize);
size_t ZSTD_compressContinue(ZSTD_CCtx *cctx, void *dst, size_t dstCapacity,
const void *src, size_t srcSize);
size_t ZSTD_compressEnd(ZSTD_CCtx *cctx, void *dst, size_t dstCapacity,
const void *src, size_t srcSize);
/*-*****************************************************************************
* Buffer-less streaming decompression (synchronous mode)
*
* A ZSTD_DCtx object is required to track streaming operations.
* Use ZSTD_initDCtx() to initialize a context.
* A ZSTD_DCtx object can be re-used multiple times.
*
* First typical operation is to retrieve frame parameters, using
* ZSTD_getFrameParams(). It fills a ZSTD_frameParams structure which provide
* important information to correctly decode the frame, such as the minimum
* rolling buffer size to allocate to decompress data (`windowSize`), and the
* dictionary ID used.
* Note: content size is optional, it may not be present. 0 means unknown.
* Note that these values could be wrong, either because of data malformation,
ext/zstd/contrib/linux-kernel/include/linux/zstd.h view on Meta::CPAN
* For skippable frames ZSTD_decompressContinue() always returns 0.
* For skippable frames ZSTD_getFrameParams() returns fparamsPtr->windowLog==0
* what means that a frame is skippable.
* Note: If fparamsPtr->frameContentSize==0, it is ambiguous: the frame might
* actually be a zstd encoded frame with no content. For purposes of
* decompression, it is valid in both cases to skip the frame using
* ZSTD_findFrameCompressedSize() to find its size in bytes.
* It also returns frame size as fparamsPtr->frameContentSize.
******************************************************************************/
/*===== Buffer-less streaming decompression functions =====*/
size_t ZSTD_decompressBegin(ZSTD_DCtx *dctx);
size_t ZSTD_decompressBegin_usingDict(ZSTD_DCtx *dctx, const void *dict,
size_t dictSize);
void ZSTD_copyDCtx(ZSTD_DCtx *dctx, const ZSTD_DCtx *preparedDCtx);
size_t ZSTD_nextSrcSizeToDecompress(ZSTD_DCtx *dctx);
size_t ZSTD_decompressContinue(ZSTD_DCtx *dctx, void *dst, size_t dstCapacity,
const void *src, size_t srcSize);
typedef enum {
ZSTDnit_frameHeader,
ZSTDnit_blockHeader,
ext/zstd/contrib/seekable_format/zstd_seekable.h view on Meta::CPAN
* each compressed individually so that decompression of a section in the
* middle of an archive only requires zstd to decompress at most a frame's
* worth of extra data, instead of the entire archive.
******************************************************************************/
typedef struct ZSTD_seekable_CStream_s ZSTD_seekable_CStream;
typedef struct ZSTD_seekable_s ZSTD_seekable;
/*-****************************************************************************
* Seekable compression - HowTo
* A ZSTD_seekable_CStream object is required to tracking streaming operation.
* Use ZSTD_seekable_createCStream() and ZSTD_seekable_freeCStream() to create/
* release resources.
*
* Streaming objects are reusable to avoid allocation and deallocation,
* to start a new compression operation call ZSTD_seekable_initCStream() on the
* compressor.
*
* Data streamed to the seekable compressor will automatically be split into
* frames of size `maxFrameSize` (provided in ZSTD_seekable_initCStream()),
* or if none is provided, will be cut off whenever ZSTD_seekable_endFrame() is
ext/zstd/contrib/seekable_format/zstdseek_compress.c view on Meta::CPAN
U32 checksum;
} framelogEntry_t;
struct ZSTD_frameLog_s {
framelogEntry_t* entries;
U32 size;
U32 capacity;
int checksumFlag;
/* for use when streaming out the seek table */
U32 seekTablePos;
U32 seekTableIndex;
} framelog_t;
struct ZSTD_seekable_CStream_s {
ZSTD_CStream* cstream;
ZSTD_frameLog framelog;
U32 frameCSize;
U32 frameDSize;
ext/zstd/doc/educational_decoder/README.md view on Meta::CPAN
Educational Decoder
===================
`zstd_decompress.c` is a self-contained implementation in C99 of a decoder,
according to the [Zstandard format specification].
While it does not implement as many features as the reference decoder,
such as the streaming API or content checksums, it is written to be easy to
follow and understand, to help understand how the Zstandard format works.
It's laid out to match the [format specification],
so it can be used to understand how complex segments could be implemented.
It also contains implementations of Huffman and FSE table decoding.
[Zstandard format specification]: https://github.com/facebook/zstd/blob/dev/doc/zstd_compression_format.md
[format specification]: https://github.com/facebook/zstd/blob/dev/doc/zstd_compression_format.md
`harness.c` provides a simple test harness around the decoder:
harness <input-file> <output-file> [dictionary]
As an additional resource to be used with this decoder,
see the `decodecorpus` tool in the [tests] directory.
It generates valid Zstandard frames that can be used to verify
a Zstandard decoder implementation.
Note that to use the tool to verify this decoder implementation,
the --content-size flag should be set,
as this decoder does not handle streaming decoding,
and so it must know the decompressed size in advance.
[tests]: https://github.com/facebook/zstd/blob/dev/tests/
ext/zstd/doc/educational_decoder/zstd_decompress.c view on Meta::CPAN
}
}
/// Decompress the data from a frame block by block
static void decompress_data(frame_context_t *const ctx, ostream_t *const out,
istream_t *const in) {
// "A frame encapsulates one or multiple blocks. Each block can be
// compressed or not, and has a guaranteed maximum content size, which
// depends on frame parameters. Unlike frames, each block depends on
// previous blocks for proper decoding. However, each block can be
// decompressed without waiting for its successor, allowing streaming
// operations."
int last_block = 0;
do {
// "Last_Block
//
// The lowest bit signals if this block is the last one. Frame ends
// right after this block.
//
// Block_Type and Block_Size
//
ext/zstd/doc/zstd_compression_format.md view on Meta::CPAN
0.3.2 (17/07/19)
Introduction
------------
The purpose of this document is to define a lossless compressed data format,
that is independent of CPU type, operating system,
file system and character set, suitable for
file compression, pipe and streaming compression,
using the [Zstandard algorithm](http://www.zstandard.org).
The text of the specification assumes a basic background in programming
at the level of bits and other primitive data representations.
The data can be produced or consumed,
even for an arbitrarily long sequentially presented input data stream,
using only an a priori bounded amount of intermediate storage,
and hence can be used in data communications.
The format uses the Zstandard compression method,
and optional [xxHash-64 checksum method](http://www.xxhash.org),
ext/zstd/doc/zstd_compression_format.md view on Meta::CPAN
Content compressed by Zstandard is transformed into a Zstandard __frame__.
Multiple frames can be appended into a single file or stream.
A frame is completely independent, has a defined beginning and end,
and a set of parameters which tells the decoder how to decompress it.
A frame encapsulates one or multiple __blocks__.
Each block contains arbitrary content, which is described by its header,
and has a guaranteed maximum content size, which depends on frame parameters.
Unlike frames, each block depends on previous blocks for proper decoding.
However, each block can be decompressed without waiting for its successor,
allowing streaming operations.
Overview
---------
- [Frames](#frames)
- [Zstandard frames](#zstandard-frames)
- [Blocks](#blocks)
- [Literals Section](#literals-section)
- [Sequences Section](#sequences-section)
- [Sequence Execution](#sequence-execution)
- [Skippable frames](#skippable-frames)
ext/zstd/doc/zstd_manual.html view on Meta::CPAN
<li><a href="#Chapter9">Streaming decompression - HowTo</a></li>
<li><a href="#Chapter10">Simple dictionary API</a></li>
<li><a href="#Chapter11">Bulk processing dictionary API</a></li>
<li><a href="#Chapter12">Dictionary helper functions</a></li>
<li><a href="#Chapter13">Advanced dictionary and prefix API</a></li>
<li><a href="#Chapter14">experimental API (static linking only)</a></li>
<li><a href="#Chapter15">Frame size functions</a></li>
<li><a href="#Chapter16">Memory management</a></li>
<li><a href="#Chapter17">Advanced compression functions</a></li>
<li><a href="#Chapter18">Advanced decompression functions</a></li>
<li><a href="#Chapter19">Advanced streaming functions</a></li>
<li><a href="#Chapter20">Buffer-less and synchronous inner streaming functions</a></li>
<li><a href="#Chapter21">Buffer-less streaming compression (synchronous mode)</a></li>
<li><a href="#Chapter22">Buffer-less streaming decompression (synchronous mode)</a></li>
<li><a href="#Chapter23">Block level API</a></li>
</ol>
<hr>
<a name="Chapter1"></a><h2>Introduction</h2><pre>
zstd, short for Zstandard, is a fast lossless compression algorithm, targeting
real-time compression scenarios at zlib-level and better compression ratios.
The zstd compression library provides in-memory compression and decompression
functions.
The library supports regular compression levels from 1 up to ZSTD_maxCLevel(),
ext/zstd/doc/zstd_manual.html view on Meta::CPAN
</b><p> Compresses `src` content as a single zstd compressed frame into already allocated `dst`.
Hint : compression runs faster if `dstCapacity` >= `ZSTD_compressBound(srcSize)`.
@return : compressed size written into `dst` (<= `dstCapacity),
or an error code if it fails (which can be tested using ZSTD_isError()).
</p></pre><BR>
<pre><b>size_t ZSTD_decompress( void* dst, size_t dstCapacity,
const void* src, size_t compressedSize);
</b><p> `compressedSize` : must be the _exact_ size of some number of compressed and/or skippable frames.
`dstCapacity` is an upper bound of originalSize to regenerate.
If user cannot imply a maximum upper bound, it's better to use streaming mode to decompress data.
@return : the number of bytes decompressed into `dst` (<= `dstCapacity`),
or an errorCode if it fails (which can be tested using ZSTD_isError()).
</p></pre><BR>
<pre><b>#define ZSTD_CONTENTSIZE_UNKNOWN (0ULL - 1)
#define ZSTD_CONTENTSIZE_ERROR (0ULL - 2)
unsigned long long ZSTD_getFrameContentSize(const void *src, size_t srcSize);
</b><p> `src` should point to the start of a ZSTD encoded frame.
`srcSize` must be at least as large as the frame header.
hint : any size >= `ZSTD_frameHeaderSize_max` is large enough.
@return : - decompressed size of `src` frame content, if known
- ZSTD_CONTENTSIZE_UNKNOWN if the size cannot be determined
- ZSTD_CONTENTSIZE_ERROR if an error occurred (e.g. invalid magic number, srcSize too small)
note 1 : a 0 return value means the frame is valid but "empty".
note 2 : decompressed size is an optional field, it may not be present, typically in streaming mode.
When `return==ZSTD_CONTENTSIZE_UNKNOWN`, data to decompress could be any size.
In which case, it's necessary to use streaming mode to decompress data.
Optionally, application can rely on some implicit limit,
as ZSTD_decompress() only needs an upper bound of decompressed size.
(For example, data could be necessarily cut into blocks <= 16 KB).
note 3 : decompressed size is always present when compression is completed using single-pass functions,
such as ZSTD_compress(), ZSTD_compressCCtx() ZSTD_compress_usingDict() or ZSTD_compress_usingCDict().
note 4 : decompressed size can be very large (64-bits value),
potentially larger than what local system can handle as a single memory segment.
In which case, it's necessary to use streaming mode to decompress data.
note 5 : If source is untrusted, decompressed size could be wrong or intentionally modified.
Always ensure return value fits within application's authorized limits.
Each application can set its own limits.
note 6 : This function replaces ZSTD_getDecompressedSize()
</p></pre><BR>
<pre><b>unsigned long long ZSTD_getDecompressedSize(const void* src, size_t srcSize);
</b><p> NOTE: This function is now obsolete, in favor of ZSTD_getFrameContentSize().
Both functions work the same way, but ZSTD_getDecompressedSize() blends
"empty", "unknown" and "error" results to the same return value (0),
ext/zstd/doc/zstd_manual.html view on Meta::CPAN
* for more info (superseded-by-cdict). */
ZSTD_c_compressionLevel=100, </b>/* Update all compression parameters according to pre-defined cLevel table<b>
* Default level is ZSTD_CLEVEL_DEFAULT==3.
* Special: value 0 means default, which is controlled by ZSTD_CLEVEL_DEFAULT.
* Note 1 : it's possible to pass a negative compression level.
* Note 2 : setting a level sets all default values of other compression parameters */
ZSTD_c_windowLog=101, </b>/* Maximum allowed back-reference distance, expressed as power of 2.<b>
* Must be clamped between ZSTD_WINDOWLOG_MIN and ZSTD_WINDOWLOG_MAX.
* Special: value 0 means "use default windowLog".
* Note: Using a windowLog greater than ZSTD_WINDOWLOG_LIMIT_DEFAULT
* requires explicitly allowing such window size at decompression stage if using streaming. */
ZSTD_c_hashLog=102, </b>/* Size of the initial probe table, as a power of 2.<b>
* Resulting memory usage is (1 << (hashLog+2)).
* Must be clamped between ZSTD_HASHLOG_MIN and ZSTD_HASHLOG_MAX.
* Larger tables improve compression ratio of strategies <= dFast,
* and improve speed of strategies > dFast.
* Special: value 0 means "use default hashLog". */
ZSTD_c_chainLog=103, </b>/* Size of the multi-probe search table, as a power of 2.<b>
* Resulting memory usage is (1 << (chainLog+2)).
* Must be clamped between ZSTD_CHAINLOG_MIN and ZSTD_CHAINLOG_MAX.
* Larger tables result in better and slower compression.
ext/zstd/doc/zstd_manual.html view on Meta::CPAN
* Must be clamped between 0 and (ZSTD_WINDOWLOG_MAX - ZSTD_HASHLOG_MIN).
* Default is MAX(0, (windowLog - ldmHashLog)), optimizing hash table usage.
* Larger values improve compression speed.
* Deviating far from default value will likely result in a compression ratio decrease.
* Special: value 0 means "automatically determine hashRateLog". */
</b>/* frame parameters */<b>
ZSTD_c_contentSizeFlag=200, </b>/* Content size will be written into frame header _whenever known_ (default:1)<b>
* Content size must be known at the beginning of compression.
* This is automatically the case when using ZSTD_compress2(),
* For streaming variants, content size must be provided with ZSTD_CCtx_setPledgedSrcSize() */
ZSTD_c_checksumFlag=201, </b>/* A 32-bits checksum of content is written at end of frame (default:0) */<b>
ZSTD_c_dictIDFlag=202, </b>/* When applicable, dictionary's ID is written into frame header (default:1) */<b>
</b>/* multi-threading parameters */<b>
</b>/* These parameters are only useful if multi-threading is enabled (compiled with build macro ZSTD_MULTITHREAD).<b>
* They return an error otherwise. */
ZSTD_c_nbWorkers=400, </b>/* Select how many threads will be spawned to compress in parallel.<b>
* When nbWorkers >= 1, triggers asynchronous mode when used with ZSTD_compressStream*() :
* ZSTD_compressStream*() consumes input and flush output if possible, but immediately gives back control to caller,
* while compression work is performed in parallel, within worker threads.
ext/zstd/doc/zstd_manual.html view on Meta::CPAN
@return : compressed size written into `dst` (<= `dstCapacity),
or an error code if it fails (which can be tested using ZSTD_isError()).
</p></pre><BR>
<a name="Chapter6"></a><h2>Advanced decompression API</h2><pre></pre>
<pre><b>typedef enum {
ZSTD_d_windowLogMax=100, </b>/* Select a size limit (in power of 2) beyond which<b>
* the streaming API will refuse to allocate memory buffer
* in order to protect the host from unreasonable memory requirements.
* This parameter is only useful in streaming mode, since no internal buffer is allocated in single-pass mode.
* By default, a decompression context accepts window sizes <= (1 << ZSTD_WINDOWLOG_LIMIT_DEFAULT).
* Special: value 0 means "use default maximum windowLog". */
</b>/* note : additional experimental parameters are also available<b>
* within the experimental section of the API.
* At the time of this writing, they include :
* ZSTD_c_format
* Because they are not stable, it's necessary to define ZSTD_STATIC_LINKING_ONLY to access them.
* note : never ever use experimentalParam? names directly
*/
ext/zstd/doc/zstd_manual.html view on Meta::CPAN
size_t pos; </b>/**< position where reading stopped. Will be updated. Necessarily 0 <= pos <= size */<b>
} ZSTD_inBuffer;
</b></pre><BR>
<pre><b>typedef struct ZSTD_outBuffer_s {
void* dst; </b>/**< start of output buffer */<b>
size_t size; </b>/**< size of output buffer */<b>
size_t pos; </b>/**< position where writing stopped. Will be updated. Necessarily 0 <= pos <= size */<b>
} ZSTD_outBuffer;
</b></pre><BR>
<a name="Chapter8"></a><h2>Streaming compression - HowTo</h2><pre>
A ZSTD_CStream object is required to track streaming operation.
Use ZSTD_createCStream() and ZSTD_freeCStream() to create/release resources.
ZSTD_CStream objects can be reused multiple times on consecutive compression operations.
It is recommended to re-use ZSTD_CStream since it will play nicer with system's memory, by re-using already allocated memory.
For parallel execution, use one separate ZSTD_CStream per thread.
note : since v1.3.0, ZSTD_CStream and ZSTD_CCtx are the same thing.
Parameters are sticky : when starting a new compression on the same context,
it will re-use the same sticky parameters as previous compression session.
ext/zstd/doc/zstd_manual.html view on Meta::CPAN
</b>/*! Equivalent to ZSTD_compressStream2(zcs, output, &emptyInput, ZSTD_e_end). */<b>
size_t ZSTD_endStream(ZSTD_CStream* zcs, ZSTD_outBuffer* output);
</b><p>
ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only);
ZSTD_CCtx_refCDict(zcs, NULL); // clear the dictionary (if any)
ZSTD_CCtx_setParameter(zcs, ZSTD_c_compressionLevel, compressionLevel);
</p></pre><BR>
<a name="Chapter9"></a><h2>Streaming decompression - HowTo</h2><pre>
A ZSTD_DStream object is required to track streaming operations.
Use ZSTD_createDStream() and ZSTD_freeDStream() to create/release resources.
ZSTD_DStream objects can be re-used multiple times.
Use ZSTD_initDStream() to start a new decompression operation.
@return : recommended first input size
Alternatively, use advanced API to set specific properties.
Use ZSTD_decompressStream() repetitively to consume your input.
The function will update both `pos` fields.
If `input.pos < input.size`, some input has not been consumed.
ext/zstd/doc/zstd_manual.html view on Meta::CPAN
<a name="Chapter15"></a><h2>Frame size functions</h2><pre></pre>
<pre><b>unsigned long long ZSTD_findDecompressedSize(const void* src, size_t srcSize);
</b><p> `src` should point to the start of a series of ZSTD encoded and/or skippable frames
`srcSize` must be the _exact_ size of this series
(i.e. there should be a frame boundary at `src + srcSize`)
@return : - decompressed size of all data in all successive frames
- if the decompressed size cannot be determined: ZSTD_CONTENTSIZE_UNKNOWN
- if an error occurred: ZSTD_CONTENTSIZE_ERROR
note 1 : decompressed size is an optional field, that may not be present, especially in streaming mode.
When `return==ZSTD_CONTENTSIZE_UNKNOWN`, data to decompress could be any size.
In which case, it's necessary to use streaming mode to decompress data.
note 2 : decompressed size is always present when compression is done with ZSTD_compress()
note 3 : decompressed size can be very large (64-bits value),
potentially larger than what local system can handle as a single memory segment.
In which case, it's necessary to use streaming mode to decompress data.
note 4 : If source is untrusted, decompressed size could be wrong or intentionally modified.
Always ensure result fits within application's authorized limits.
Each application can set its own limits.
note 5 : ZSTD_findDecompressedSize handles multiple frames, and so it must traverse the input to
read each contained frame header. This is fast as most of the data is skipped,
however it does mean that all frame data must be present and valid.
</p></pre><BR>
<pre><b>unsigned long long ZSTD_decompressBound(const void* src, size_t srcSize);
</b><p> `src` should point to the start of a series of ZSTD encoded and/or skippable frames
ext/zstd/doc/zstd_manual.html view on Meta::CPAN
size_t ZSTD_estimateDStreamSize_fromFrame(const void* src, size_t srcSize);
</b><p> ZSTD_estimateCStreamSize() will provide a budget large enough for any compression level up to selected one.
It will also consider src size to be arbitrarily "large", which is worst case.
If srcSize is known to always be small, ZSTD_estimateCStreamSize_usingCParams() can provide a tighter estimation.
ZSTD_estimateCStreamSize_usingCParams() can be used in tandem with ZSTD_getCParams() to create cParams from compressionLevel.
ZSTD_estimateCStreamSize_usingCCtxParams() can be used in tandem with ZSTD_CCtxParams_setParameter(). Only single-threaded compression is supported. This function will return an error code if ZSTD_c_nbWorkers is >= 1.
Note : CStream size estimation is only correct for single-threaded compression.
ZSTD_DStream memory budget depends on window Size.
This information can be passed manually, using ZSTD_estimateDStreamSize,
or deducted from a valid frame Header, using ZSTD_estimateDStreamSize_fromFrame();
Note : if streaming is init with function ZSTD_init?Stream_usingDict(),
an internal ?Dict will be created, which additional size is not estimated here.
In this case, get total size by adding ZSTD_estimate?DictSize
</p></pre><BR>
<pre><b>size_t ZSTD_estimateCDictSize(size_t dictSize, int compressionLevel);
size_t ZSTD_estimateCDictSize_advanced(size_t dictSize, ZSTD_compressionParameters cParams, ZSTD_dictLoadMethod_e dictLoadMethod);
size_t ZSTD_estimateDDictSize(size_t dictSize, ZSTD_dictLoadMethod_e dictLoadMethod);
</b><p> ZSTD_estimateCDictSize() will bet that src size is relatively "small", and content is copied, like ZSTD_createCDict().
ZSTD_estimateCDictSize_advanced() makes it possible to control compression parameters precisely, like ZSTD_createCDict_advanced().
Note : dictionaries created by reference (`ZSTD_dlm_byRef`) are logically smaller.
ext/zstd/doc/zstd_manual.html view on Meta::CPAN
</p></pre><BR>
<pre><b>size_t ZSTD_DCtx_refPrefix_advanced(ZSTD_DCtx* dctx, const void* prefix, size_t prefixSize, ZSTD_dictContentType_e dictContentType);
</b><p> Same as ZSTD_DCtx_refPrefix(), but gives finer control over
how to interpret prefix content (automatic ? force raw mode (default) ? full mode only ?)
</p></pre><BR>
<pre><b>size_t ZSTD_DCtx_setMaxWindowSize(ZSTD_DCtx* dctx, size_t maxWindowSize);
</b><p> Refuses allocating internal buffers for frames requiring a window size larger than provided limit.
This protects a decoder context from reserving too much memory for itself (potential attack scenario).
This parameter is only useful in streaming mode, since no internal buffer is allocated in single-pass mode.
By default, a decompression context accepts all window sizes <= (1 << ZSTD_WINDOWLOG_LIMIT_DEFAULT)
@return : 0, or an error code (which can be tested using ZSTD_isError()).
</p></pre><BR>
<pre><b>size_t ZSTD_DCtx_setFormat(ZSTD_DCtx* dctx, ZSTD_format_e format);
</b><p> Instruct the decoder context about what kind of data to decode next.
This instruction is mandatory to decode data without a fully-formed header,
such ZSTD_f_zstd1_magicless for example.
@return : 0, or an error code (which can be tested using ZSTD_isError()).
ext/zstd/doc/zstd_manual.html view on Meta::CPAN
ZSTD_DCtx* dctx,
void* dst, size_t dstCapacity, size_t* dstPos,
const void* src, size_t srcSize, size_t* srcPos);
</b><p> Same as ZSTD_decompressStream(),
but using only integral types as arguments.
This can be helpful for binders from dynamic languages
which have troubles handling structures containing memory pointers.
</p></pre><BR>
<a name="Chapter19"></a><h2>Advanced streaming functions</h2><pre> Warning : most of these functions are now redundant with the Advanced API.
Once Advanced API reaches "stable" status,
redundant functions will be deprecated, and then at some point removed.
<BR></pre>
<h3>Advanced Streaming compression functions</h3><pre></pre><b><pre></b>/**! ZSTD_initCStream_srcSize() :<b>
* This function is deprecated, and equivalent to:
* ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only);
* ZSTD_CCtx_refCDict(zcs, NULL); // clear the dictionary (if any)
* ZSTD_CCtx_setParameter(zcs, ZSTD_c_compressionLevel, compressionLevel);
* ZSTD_CCtx_setPledgedSrcSize(zcs, pledgedSrcSize);
ext/zstd/doc/zstd_manual.html view on Meta::CPAN
size_t ZSTD_initDStream_usingDDict(ZSTD_DStream* zds, const ZSTD_DDict* ddict);
</b>/**<b>
* This function is deprecated, and is equivalent to:
*
* ZSTD_DCtx_reset(zds, ZSTD_reset_session_only);
*
* re-use decompression parameters from previous init; saves dictionary loading
*/
size_t ZSTD_resetDStream(ZSTD_DStream* zds);
</pre></b><BR>
<a name="Chapter20"></a><h2>Buffer-less and synchronous inner streaming functions</h2><pre>
This is an advanced API, giving full control over buffer management, for users which need direct control over memory.
But it's also a complex one, with several restrictions, documented below.
Prefer normal streaming API for an easier experience.
<BR></pre>
<a name="Chapter21"></a><h2>Buffer-less streaming compression (synchronous mode)</h2><pre>
A ZSTD_CCtx object is required to track streaming operations.
Use ZSTD_createCCtx() / ZSTD_freeCCtx() to manage resource.
ZSTD_CCtx object can be re-used multiple times within successive compression operations.
Start by initializing a context.
Use ZSTD_compressBegin(), or ZSTD_compressBegin_usingDict() for dictionary compression,
or ZSTD_compressBegin_advanced(), for finer parameter control.
It's also possible to duplicate a reference context which has already been initialized, using ZSTD_copyCCtx()
Then, consume your input using ZSTD_compressContinue().
There are some important considerations to keep in mind when using this advanced function :
ext/zstd/doc/zstd_manual.html view on Meta::CPAN
- ZSTD_compressContinue() detects that prior input has been overwritten when `src` buffer overlaps.
In which case, it will "discard" the relevant memory section from its history.
Finish a frame with ZSTD_compressEnd(), which will write the last block(s) and optional checksum.
It's possible to use srcSize==0, in which case, it will write a final empty block to end the frame.
Without last block mark, frames are considered unfinished (hence corrupted) by compliant decoders.
`ZSTD_CCtx` object can be re-used (ZSTD_compressBegin()) to compress again.
<BR></pre>
<h3>Buffer-less streaming compression functions</h3><pre></pre><b><pre>size_t ZSTD_compressBegin(ZSTD_CCtx* cctx, int compressionLevel);
size_t ZSTD_compressBegin_usingDict(ZSTD_CCtx* cctx, const void* dict, size_t dictSize, int compressionLevel);
size_t ZSTD_compressBegin_advanced(ZSTD_CCtx* cctx, const void* dict, size_t dictSize, ZSTD_parameters params, unsigned long long pledgedSrcSize); </b>/**< pledgedSrcSize : If srcSize is not known at init time, use ZSTD_CONTENTSIZE_UNKNOWN */<b>
size_t ZSTD_compressBegin_usingCDict(ZSTD_CCtx* cctx, const ZSTD_CDict* cdict); </b>/**< note: fails if cdict==NULL */<b>
size_t ZSTD_compressBegin_usingCDict_advanced(ZSTD_CCtx* const cctx, const ZSTD_CDict* const cdict, ZSTD_frameParameters const fParams, unsigned long long const pledgedSrcSize); </b>/* compression parameters are already set within cdict. pledgedSrc...
size_t ZSTD_copyCCtx(ZSTD_CCtx* cctx, const ZSTD_CCtx* preparedCCtx, unsigned long long pledgedSrcSize); </b>/**< note: if pledgedSrcSize is not known, use ZSTD_CONTENTSIZE_UNKNOWN */<b>
</pre></b><BR>
<a name="Chapter22"></a><h2>Buffer-less streaming decompression (synchronous mode)</h2><pre>
A ZSTD_DCtx object is required to track streaming operations.
Use ZSTD_createDCtx() / ZSTD_freeDCtx() to manage it.
A ZSTD_DCtx object can be re-used multiple times.
First typical operation is to retrieve frame parameters, using ZSTD_getFrameHeader().
Frame header is extracted from the beginning of compressed frame, so providing only the frame's beginning is enough.
Data fragment must be large enough to ensure successful decoding.
`ZSTD_frameHeaderSize_max` bytes is guaranteed to always be large enough.
@result : 0 : successful decoding, the `ZSTD_frameHeader` structure is correctly filled.
>0 : `srcSize` is too small, please provide at least @result bytes on next attempt.
errorCode, which can be tested using ZSTD_isError().
ext/zstd/doc/zstd_manual.html view on Meta::CPAN
Skippable frames allow integration of user-defined data into a flow of concatenated frames.
Skippable frames will be ignored (skipped) by decompressor.
The format of skippable frames is as follows :
a) Skippable frame ID - 4 Bytes, Little endian format, any value from 0x184D2A50 to 0x184D2A5F
b) Frame Size - 4 Bytes, Little endian format, unsigned 32-bits
c) Frame Content - any content (User Data) of length equal to Frame Size
For skippable frames ZSTD_getFrameHeader() returns zfhPtr->frameType==ZSTD_skippableFrame.
For skippable frames ZSTD_decompressContinue() always returns 0 : it only skips the content.
<BR></pre>
<h3>Buffer-less streaming decompression functions</h3><pre></pre><b><pre>typedef enum { ZSTD_frame, ZSTD_skippableFrame } ZSTD_frameType_e;
typedef struct {
unsigned long long frameContentSize; </b>/* if == ZSTD_CONTENTSIZE_UNKNOWN, it means this field is not available. 0 means "empty" */<b>
unsigned long long windowSize; </b>/* can be very large, up to <= frameContentSize */<b>
unsigned blockSizeMax;
ZSTD_frameType_e frameType; </b>/* if == ZSTD_skippableFrame, frameContentSize is the size of skippable content */<b>
unsigned headerSize;
unsigned dictID;
unsigned checksumFlag;
} ZSTD_frameHeader;
</pre></b><BR>
ext/zstd/examples/Makefile view on Meta::CPAN
CPPFLAGS += -I../lib
LIB = ../lib/libzstd.a
.PHONY: default all clean test
default: all
all: simple_compression simple_decompression \
multiple_simple_compression\
dictionary_compression dictionary_decompression \
streaming_compression streaming_decompression \
multiple_streaming_compression streaming_memory_usage
$(LIB) :
$(MAKE) -C ../lib libzstd.a
simple_compression : simple_compression.c common.h $(LIB)
$(CC) $(CPPFLAGS) $(CFLAGS) $< $(LIB) $(LDFLAGS) -o $@
simple_decompression : simple_decompression.c common.h $(LIB)
$(CC) $(CPPFLAGS) $(CFLAGS) $< $(LIB) $(LDFLAGS) -o $@
multiple_simple_compression : multiple_simple_compression.c common.h $(LIB)
$(CC) $(CPPFLAGS) $(CFLAGS) $< $(LIB) $(LDFLAGS) -o $@
dictionary_compression : dictionary_compression.c common.h $(LIB)
$(CC) $(CPPFLAGS) $(CFLAGS) $< $(LIB) $(LDFLAGS) -o $@
dictionary_decompression : dictionary_decompression.c common.h $(LIB)
$(CC) $(CPPFLAGS) $(CFLAGS) $< $(LIB) $(LDFLAGS) -o $@
streaming_compression : streaming_compression.c common.h $(LIB)
$(CC) $(CPPFLAGS) $(CFLAGS) $< $(LIB) $(LDFLAGS) -o $@
multiple_streaming_compression : multiple_streaming_compression.c common.h $(LIB)
$(CC) $(CPPFLAGS) $(CFLAGS) $< $(LIB) $(LDFLAGS) -o $@
streaming_decompression : streaming_decompression.c common.h $(LIB)
$(CC) $(CPPFLAGS) $(CFLAGS) $< $(LIB) $(LDFLAGS) -o $@
streaming_memory_usage : streaming_memory_usage.c $(LIB)
$(CC) $(CPPFLAGS) $(CFLAGS) $< $(LIB) $(LDFLAGS) -o $@
clean:
@rm -f core *.o tmp* result* *.zst \
simple_compression simple_decompression \
multiple_simple_compression \
dictionary_compression dictionary_decompression \
streaming_compression streaming_decompression \
multiple_streaming_compression streaming_memory_usage
@echo Cleaning completed
test: all
cp README.md tmp
cp Makefile tmp2
@echo -- Simple compression tests
./simple_compression tmp
./simple_decompression tmp.zst
./multiple_simple_compression *.c
./streaming_decompression tmp.zst > /dev/null
@echo -- Streaming memory usage
./streaming_memory_usage
@echo -- Streaming compression tests
./streaming_compression tmp
./streaming_decompression tmp.zst > /dev/null
@echo -- Edge cases detection
! ./streaming_decompression tmp # invalid input, must fail
! ./simple_decompression tmp # invalid input, must fail
touch tmpNull # create 0-size file
./simple_compression tmpNull
./simple_decompression tmpNull.zst # 0-size frame : must work
@echo -- Multiple streaming tests
./multiple_streaming_compression *.c
@echo -- Dictionary compression tests
./dictionary_compression tmp2 tmp README.md
./dictionary_decompression tmp2.zst tmp.zst README.md
$(RM) tmp* *.zst
@echo tests completed
ext/zstd/examples/README.md view on Meta::CPAN
Only compatible with simple compression.
Result remains in memory.
Introduces usage of : `ZSTD_decompress()`
- [Multiple simple compression](multiple_simple_compression.c) :
Compress multiple files (in simple mode) in a single command line.
Demonstrates memory preservation technique that
minimizes malloc()/free() calls by re-using existing resources.
Introduces usage of : `ZSTD_compressCCtx()`
- [Streaming memory usage](streaming_memory_usage.c) :
Provides amount of memory used by streaming context.
Introduces usage of : `ZSTD_sizeof_CStream()`
- [Streaming compression](streaming_compression.c) :
Compress a single file.
Introduces usage of : `ZSTD_compressStream()`
- [Multiple Streaming compression](multiple_streaming_compression.c) :
Compress multiple files (in streaming mode) in a single command line.
Introduces memory usage preservation technique,
reducing impact of malloc()/free() and memset() by re-using existing resources.
- [Streaming decompression](streaming_decompression.c) :
Decompress a single file compressed by zstd.
Compatible with both simple and streaming compression.
Result is sent to stdout.
Introduces usage of : `ZSTD_decompressStream()`
- [Dictionary compression](dictionary_compression.c) :
Compress multiple files using the same dictionary.
Introduces usage of : `ZSTD_createCDict()` and `ZSTD_compress_usingCDict()`
- [Dictionary decompression](dictionary_decompression.c) :
Decompress multiple files using the same dictionary.
Result remains in memory.
ext/zstd/examples/dictionary_decompression.c view on Meta::CPAN
return ddict;
}
static void decompress(const char* fname, const ZSTD_DDict* ddict)
{
size_t cSize;
void* const cBuff = mallocAndLoadFile_orDie(fname, &cSize);
/* Read the content size from the frame header. For simplicity we require
* that it is always present. By default, zstd will write the content size
* in the header when it is known. If you can't guarantee that the frame
* content size is always written into the header, either use streaming
* decompression, or ZSTD_decompressBound().
*/
unsigned long long const rSize = ZSTD_getFrameContentSize(cBuff, cSize);
CHECK(rSize != ZSTD_CONTENTSIZE_ERROR, "%s: not compressed by zstd!", fname);
CHECK(rSize != ZSTD_CONTENTSIZE_UNKNOWN, "%s: original size unknown!", fname);
void* const rBuff = malloc_orDie((size_t)rSize);
/* Check that the dictionary ID matches.
* If a non-zstd dictionary is used, then both will be zero.
* By default zstd always writes the dictionary ID into the frame.
ext/zstd/examples/multiple_streaming_compression.c view on Meta::CPAN
/* Reset the context to a clean state to start a new compression operation.
* The parameters are sticky, so we keep the compression level and extra
* parameters that we set in createResources_orDie().
*/
CHECK_ZSTD( ZSTD_CCtx_reset(ress.cctx, ZSTD_reset_session_only) );
size_t const toRead = ress.buffInSize;
size_t read;
while ( (read = fread_orDie(ress.buffIn, toRead, fin)) ) {
/* This loop is the same as streaming_compression.c.
* See that file for detailed comments.
*/
int const lastChunk = (read < toRead);
ZSTD_EndDirective const mode = lastChunk ? ZSTD_e_end : ZSTD_e_continue;
ZSTD_inBuffer input = { ress.buffIn, read, 0 };
int finished;
do {
ZSTD_outBuffer output = { ress.buffOut, ress.buffOutSize, 0 };
size_t const remaining = ZSTD_compressStream2(ress.cctx, &output, &input, mode);
ext/zstd/examples/simple_decompression.c view on Meta::CPAN
#include <zstd.h> // presumes zstd library is installed
#include "common.h" // Helper functions, CHECK(), and CHECK_ZSTD()
static void decompress(const char* fname)
{
size_t cSize;
void* const cBuff = mallocAndLoadFile_orDie(fname, &cSize);
/* Read the content size from the frame header. For simplicity we require
* that it is always present. By default, zstd will write the content size
* in the header when it is known. If you can't guarantee that the frame
* content size is always written into the header, either use streaming
* decompression, or ZSTD_decompressBound().
*/
unsigned long long const rSize = ZSTD_getFrameContentSize(cBuff, cSize);
CHECK(rSize != ZSTD_CONTENTSIZE_ERROR, "%s: not compressed by zstd!", fname);
CHECK(rSize != ZSTD_CONTENTSIZE_UNKNOWN, "%s: original size unknown!", fname);
void* const rBuff = malloc_orDie((size_t)rSize);
/* Decompress.
* If you are doing many decompressions, you may want to reuse the context
ext/zstd/examples/streaming_decompression.c view on Meta::CPAN
void* const buffIn = malloc_orDie(buffInSize);
FILE* const fout = stdout;
size_t const buffOutSize = ZSTD_DStreamOutSize(); /* Guarantee to successfully flush at least one complete compressed block in all circumstances. */
void* const buffOut = malloc_orDie(buffOutSize);
ZSTD_DCtx* const dctx = ZSTD_createDCtx();
CHECK(dctx != NULL, "ZSTD_createDCtx() failed!");
/* This loop assumes that the input file is one or more concatenated zstd
* streams. This example won't work if there is trailing non-zstd data at
* the end, but streaming decompression in general handles this case.
* ZSTD_decompressStream() returns 0 exactly when the frame is completed,
* and doesn't consume input after the frame.
*/
size_t const toRead = buffInSize;
size_t read;
while ( (read = fread_orDie(buffIn, toRead, fin)) ) {
ZSTD_inBuffer input = { buffIn, read, 0 };
/* Given a valid frame, zstd won't consume the last byte of the frame
* until it has flushed all of the decompressed data of the frame.
* Therefore, instead of checking if the return code is 0, we can
ext/zstd/examples/streaming_memory_usage.c view on Meta::CPAN
(*stringPtr)++ ;
if (**stringPtr=='i') (*stringPtr)++;
if (**stringPtr=='B') (*stringPtr)++;
}
return result;
}
int main(int argc, char const *argv[]) {
printf("\n Zstandard (v%s) memory usage for streaming : \n\n", ZSTD_versionString());
unsigned wLog = 0;
if (argc > 1) {
const char* valStr = argv[1];
wLog = readU32FromChar(&valStr);
}
int compressionLevel;
for (compressionLevel = 1; compressionLevel <= MAX_TESTED_LEVEL; compressionLevel++) {
#define INPUT_SIZE 5
ext/zstd/lib/Makefile view on Meta::CPAN
@$(INSTALL_DATA) dictBuilder/zdict.h $(DESTDIR)$(INCLUDEDIR)
uninstall:
@$(RM) $(DESTDIR)$(LIBDIR)/libzstd.a
@$(RM) $(DESTDIR)$(LIBDIR)/libzstd.$(SHARED_EXT)
@$(RM) $(DESTDIR)$(LIBDIR)/libzstd.$(SHARED_EXT_MAJOR)
@$(RM) $(DESTDIR)$(LIBDIR)/$(LIBZSTD)
@$(RM) $(DESTDIR)$(PKGCONFIGDIR)/libzstd.pc
@$(RM) $(DESTDIR)$(INCLUDEDIR)/zstd.h
@$(RM) $(DESTDIR)$(INCLUDEDIR)/zstd_errors.h
@$(RM) $(DESTDIR)$(INCLUDEDIR)/zbuff.h # Deprecated streaming functions
@$(RM) $(DESTDIR)$(INCLUDEDIR)/zdict.h
@echo zstd libraries successfully uninstalled
endif
ext/zstd/lib/README.md view on Meta::CPAN
file it should be linked with `dll\libzstd.dll`. For example:
```
gcc $(CFLAGS) -Iinclude/ test-dll.c -o test-dll dll\libzstd.dll
```
The compiled executable will require ZSTD DLL which is available at `dll\libzstd.dll`.
#### Deprecated API
Obsolete API on their way out are stored in directory `lib/deprecated`.
At this stage, it contains older streaming prototypes, in `lib/deprecated/zbuff.h`.
These prototypes will be removed in some future version.
Consider migrating code towards supported streaming API exposed in `zstd.h`.
#### Miscellaneous
The other files are not source code. There are :
- `BUCK` : support for `buck` build system (https://buckbuild.com/)
- `Makefile` : `make` script to build and install zstd library (static and dynamic)
- `README.md` : this file
- `dll/` : resources directory for Windows compilation
ext/zstd/lib/common/xxhash.h view on Meta::CPAN
/*! State allocation, compatible with dynamic libraries */
XXH_PUBLIC_API XXH32_state_t* XXH32_createState(void);
XXH_PUBLIC_API XXH_errorcode XXH32_freeState(XXH32_state_t* statePtr);
XXH_PUBLIC_API XXH64_state_t* XXH64_createState(void);
XXH_PUBLIC_API XXH_errorcode XXH64_freeState(XXH64_state_t* statePtr);
/* hash streaming */
XXH_PUBLIC_API XXH_errorcode XXH32_reset (XXH32_state_t* statePtr, unsigned int seed);
XXH_PUBLIC_API XXH_errorcode XXH32_update (XXH32_state_t* statePtr, const void* input, size_t length);
XXH_PUBLIC_API XXH32_hash_t XXH32_digest (const XXH32_state_t* statePtr);
XXH_PUBLIC_API XXH_errorcode XXH64_reset (XXH64_state_t* statePtr, unsigned long long seed);
XXH_PUBLIC_API XXH_errorcode XXH64_update (XXH64_state_t* statePtr, const void* input, size_t length);
XXH_PUBLIC_API XXH64_hash_t XXH64_digest (const XXH64_state_t* statePtr);
/*
ext/zstd/lib/compress/zstd_compress.c view on Meta::CPAN
size_t ZSTD_estimateCStreamSize_usingCCtxParams(const ZSTD_CCtx_params* params)
{
RETURN_ERROR_IF(params->nbWorkers > 0, GENERIC, "Estimate CCtx size is supported for single-threaded compression only.");
{ ZSTD_compressionParameters const cParams =
ZSTD_getCParamsFromCCtxParams(params, 0, 0);
size_t const CCtxSize = ZSTD_estimateCCtxSize_usingCCtxParams(params);
size_t const blockSize = MIN(ZSTD_BLOCKSIZE_MAX, (size_t)1 << cParams.windowLog);
size_t const inBuffSize = ((size_t)1 << cParams.windowLog) + blockSize;
size_t const outBuffSize = ZSTD_compressBound(blockSize) + 1;
size_t const streamingSize = inBuffSize + outBuffSize;
return CCtxSize + streamingSize;
}
}
size_t ZSTD_estimateCStreamSize_usingCParams(ZSTD_compressionParameters cParams)
{
ZSTD_CCtx_params const params = ZSTD_makeCCtxParamsFromCParams(cParams);
return ZSTD_estimateCStreamSize_usingCCtxParams(¶ms);
}
static size_t ZSTD_estimateCStreamSize_internal(int compressionLevel)
ext/zstd/lib/compress/zstd_compress.c view on Meta::CPAN
}
#endif
{ ZSTD_frameProgression fp;
size_t const buffered = (cctx->inBuff == NULL) ? 0 :
cctx->inBuffPos - cctx->inToCompress;
if (buffered) assert(cctx->inBuffPos >= cctx->inToCompress);
assert(buffered <= ZSTD_BLOCKSIZE_MAX);
fp.ingested = cctx->consumedSrcSize + buffered;
fp.consumed = cctx->consumedSrcSize;
fp.produced = cctx->producedCSize;
fp.flushed = cctx->producedCSize; /* simplified; some data might still be left within streaming output buffer */
fp.currentJobID = 0;
fp.nbActiveWorkers = 0;
return fp;
} }
/*! ZSTD_toFlushNow()
* Only useful for multithreading scenarios currently (nbWorkers >= 1).
*/
size_t ZSTD_toFlushNow(ZSTD_CCtx* cctx)
{
#ifdef ZSTD_MULTITHREAD
if (cctx->appliedParams.nbWorkers > 0) {
return ZSTDMT_toFlushNow(cctx->mtctx);
}
#endif
(void)cctx;
return 0; /* over-simplification; could also check if context is currently running in streaming mode, and in which case, report how many bytes are left to be flushed within output buffer */
}
static U32 ZSTD_equivalentCParams(ZSTD_compressionParameters cParams1,
ZSTD_compressionParameters cParams2)
{
return (cParams1.hashLog == cParams2.hashLog)
& (cParams1.chainLog == cParams2.chainLog)
& (cParams1.strategy == cParams2.strategy) /* opt parser space */
ext/zstd/lib/compress/zstd_compress.c view on Meta::CPAN
(ldmParams1.enableLdm == ldmParams2.enableLdm &&
ldmParams1.hashLog == ldmParams2.hashLog &&
ldmParams1.bucketSizeLog == ldmParams2.bucketSizeLog &&
ldmParams1.minMatchLength == ldmParams2.minMatchLength &&
ldmParams1.hashRateLog == ldmParams2.hashRateLog);
}
typedef enum { ZSTDb_not_buffered, ZSTDb_buffered } ZSTD_buffered_policy_e;
/* ZSTD_sufficientBuff() :
* check internal buffers exist for streaming if buffPol == ZSTDb_buffered .
* Note : they are assumed to be correctly sized if ZSTD_equivalentCParams()==1 */
static U32 ZSTD_sufficientBuff(size_t bufferSize1, size_t maxNbSeq1,
size_t maxNbLit1,
ZSTD_buffered_policy_e buffPol2,
ZSTD_compressionParameters cParams2,
U64 pledgedSrcSize)
{
size_t const windowSize2 = MAX(1, (size_t)MIN(((U64)1 << cParams2.windowLog), pledgedSrcSize));
size_t const blockSize2 = MIN(ZSTD_BLOCKSIZE_MAX, windowSize2);
size_t const maxNbSeq2 = blockSize2 / ((cParams2.minMatch == 3) ? 3 : 4);
ext/zstd/lib/compress/zstd_compress_internal.h view on Meta::CPAN
size_t staticSize;
seqStore_t seqStore; /* sequences storage ptrs */
ldmState_t ldmState; /* long distance matching state */
rawSeq* ldmSequences; /* Storage for the ldm output sequences */
size_t maxNbLdmSequences;
rawSeqStore_t externSeqStore; /* Mutable reference to external sequences */
ZSTD_blockState_t blockState;
U32* entropyWorkspace; /* entropy workspace of HUF_WORKSPACE_SIZE bytes */
/* streaming */
char* inBuff;
size_t inBuffSize;
size_t inToCompress;
size_t inBuffPos;
size_t inBuffTarget;
char* outBuff;
size_t outBuffSize;
size_t outBuffContentSize;
size_t outBuffFlushedSize;
ZSTD_cStreamStage streamStage;
ext/zstd/lib/compress/zstd_compress_internal.h view on Meta::CPAN
* ============================================================== */
/* ZSTD_getCParamsFromCCtxParams() :
* cParams are built depending on compressionLevel, src size hints,
* LDM and manually set compression parameters.
*/
ZSTD_compressionParameters ZSTD_getCParamsFromCCtxParams(
const ZSTD_CCtx_params* CCtxParams, U64 srcSizeHint, size_t dictSize);
/*! ZSTD_initCStream_internal() :
* Private use only. Init streaming operation.
* expects params to be valid.
* must receive dict, or cdict, or none, but not both.
* @return : 0, or an error code */
size_t ZSTD_initCStream_internal(ZSTD_CStream* zcs,
const void* dict, size_t dictSize,
const ZSTD_CDict* cdict,
ZSTD_CCtx_params params, unsigned long long pledgedSrcSize);
void ZSTD_resetSeqStore(seqStore_t* ssPtr);
ext/zstd/lib/compress/zstdmt_compress.c view on Meta::CPAN
{
ZSTDMT_jobDescription* const job = (ZSTDMT_jobDescription*)jobDescription;
ZSTD_CCtx_params jobParams = job->params; /* do not modify job->params ! copy it, modify the copy */
ZSTD_CCtx* const cctx = ZSTDMT_getCCtx(job->cctxPool);
rawSeqStore_t rawSeqStore = ZSTDMT_getSeq(job->seqPool);
buffer_t dstBuff = job->dstBuff;
size_t lastCBlockSize = 0;
/* resources */
if (cctx==NULL) JOB_ERROR(ERROR(memory_allocation));
if (dstBuff.start == NULL) { /* streaming job : doesn't provide a dstBuffer */
dstBuff = ZSTDMT_getBuffer(job->bufPool);
if (dstBuff.start==NULL) JOB_ERROR(ERROR(memory_allocation));
job->dstBuff = dstBuff; /* this value can be read in ZSTDMT_flush, when it copies the whole job */
}
if (jobParams.ldmParams.enableLdm && rawSeqStore.seq == NULL)
JOB_ERROR(ERROR(memory_allocation));
/* Don't compute the checksum for chunks, since we compute it externally,
* but write it in the header.
*/
ext/zstd/lib/compress/zstdmt_compress.c view on Meta::CPAN
ZSTD_CCtx_params cctxParams = mtctx->params; /* retrieve sticky params */
DEBUGLOG(4, "ZSTDMT_initCStream (cLevel=%i)", compressionLevel);
cctxParams.cParams = params.cParams;
cctxParams.fParams = params.fParams;
return ZSTDMT_initCStream_internal(mtctx, NULL, 0, ZSTD_dct_auto, NULL, cctxParams, ZSTD_CONTENTSIZE_UNKNOWN);
}
/* ZSTDMT_writeLastEmptyBlock()
* Write a single empty block with an end-of-frame to finish a frame.
* Job must be created from streaming variant.
* This function is always successful if expected conditions are fulfilled.
*/
static void ZSTDMT_writeLastEmptyBlock(ZSTDMT_jobDescription* job)
{
assert(job->lastJob == 1);
assert(job->src.size == 0); /* last job is empty -> will be simplified into a last empty block */
assert(job->firstJob == 0); /* cannot be first job, as it also needs to create frame header */
assert(job->dstBuff.start == NULL); /* invoked from streaming variant only (otherwise, dstBuff might be user's output) */
job->dstBuff = ZSTDMT_getBuffer(job->bufPool);
if (job->dstBuff.start == NULL) {
job->cSize = ERROR(memory_allocation);
return;
}
assert(job->dstBuff.capacity >= ZSTD_blockHeaderSize); /* no buffer should ever be that small */
job->src = kNullRange;
job->cSize = ZSTD_writeLastEmptyBlock(job->dstBuff.start, job->dstBuff.capacity);
assert(!ZSTD_isError(job->cSize));
assert(job->consumed == 0);
ext/zstd/lib/compress/zstdmt_compress.h view on Meta::CPAN
void ZSTDMT_updateCParams_whileCompressing(ZSTDMT_CCtx* mtctx, const ZSTD_CCtx_params* cctxParams);
/*! ZSTDMT_getFrameProgression():
* tells how much data has been consumed (input) and produced (output) for current frame.
* able to count progression inside worker threads.
*/
ZSTD_frameProgression ZSTDMT_getFrameProgression(ZSTDMT_CCtx* mtctx);
/*! ZSTDMT_initCStream_internal() :
* Private use only. Init streaming operation.
* expects params to be valid.
* must receive dict, or cdict, or none, but not both.
* @return : 0, or an error code */
size_t ZSTDMT_initCStream_internal(ZSTDMT_CCtx* zcs,
const void* dict, size_t dictSize, ZSTD_dictContentType_e dictContentType,
const ZSTD_CDict* cdict,
ZSTD_CCtx_params params, unsigned long long pledgedSrcSize);
#if defined (__cplusplus)
ext/zstd/lib/decompress/zstd_decompress_internal.h view on Meta::CPAN
size_t staticSize;
int bmi2; /* == 1 if the CPU supports BMI2 and 0 otherwise. CPU support is determined dynamically once per context lifetime. */
/* dictionary */
ZSTD_DDict* ddictLocal;
const ZSTD_DDict* ddict; /* set by ZSTD_initDStream_usingDDict(), or ZSTD_DCtx_refDDict() */
U32 dictID;
int ddictIsCold; /* if == 1 : dictionary is "new" for working context, and presumed "cold" (not in cpu cache) */
ZSTD_dictUses_e dictUses;
/* streaming */
ZSTD_dStreamStage streamStage;
char* inBuff;
size_t inBuffSize;
size_t inPos;
size_t maxWindowSize;
char* outBuff;
size_t outBuffSize;
size_t outStart;
size_t outEnd;
size_t lhSize;
ext/zstd/lib/deprecated/zbuff.h view on Meta::CPAN
*
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
* You may select, at your option, one of the above-listed licenses.
*/
/* ***************************************************************
* NOTES/WARNINGS
******************************************************************/
/* The streaming API defined here is deprecated.
* Consider migrating towards ZSTD_compressStream() API in `zstd.h`
* See 'lib/README.md'.
*****************************************************************/
#if defined (__cplusplus)
extern "C" {
#endif
#ifndef ZSTD_BUFFERED_H_23987
ext/zstd/lib/deprecated/zbuff.h view on Meta::CPAN
# else
# pragma message("WARNING: You need to implement ZBUFF_DEPRECATED for this compiler")
# define ZBUFF_DEPRECATED(message) ZSTDLIB_API
# endif
#endif /* ZBUFF_DISABLE_DEPRECATE_WARNINGS */
/* *************************************
* Streaming functions
***************************************/
/* This is the easier "buffered" streaming API,
* using an internal buffer to lift all restrictions on user-provided buffers
* which can be any size, any place, for both input and output.
* ZBUFF and ZSTD are 100% interoperable,
* frames created by one can be decoded by the other one */
typedef ZSTD_CStream ZBUFF_CCtx;
ZBUFF_DEPRECATED("use ZSTD_createCStream") ZBUFF_CCtx* ZBUFF_createCCtx(void);
ZBUFF_DEPRECATED("use ZSTD_freeCStream") size_t ZBUFF_freeCCtx(ZBUFF_CCtx* cctx);
ZBUFF_DEPRECATED("use ZSTD_initCStream") size_t ZBUFF_compressInit(ZBUFF_CCtx* cctx, int compressionLevel);
ZBUFF_DEPRECATED("use ZSTD_initCStream_usingDict") size_t ZBUFF_compressInitDictionary(ZBUFF_CCtx* cctx, const void* dict, size_t dictSize, int compressionLevel);
ZBUFF_DEPRECATED("use ZSTD_compressStream") size_t ZBUFF_compressContinue(ZBUFF_CCtx* cctx, void* dst, size_t* dstCapacityPtr, const void* src, size_t* srcSizePtr);
ZBUFF_DEPRECATED("use ZSTD_flushStream") size_t ZBUFF_compressFlush(ZBUFF_CCtx* cctx, void* dst, size_t* dstCapacityPtr);
ZBUFF_DEPRECATED("use ZSTD_endStream") size_t ZBUFF_compressEnd(ZBUFF_CCtx* cctx, void* dst, size_t* dstCapacityPtr);
/*-*************************************************
* Streaming compression - howto
*
* A ZBUFF_CCtx object is required to track streaming operation.
* Use ZBUFF_createCCtx() and ZBUFF_freeCCtx() to create/release resources.
* ZBUFF_CCtx objects can be reused multiple times.
*
* Start by initializing ZBUF_CCtx.
* Use ZBUFF_compressInit() to start a new compression operation.
* Use ZBUFF_compressInitDictionary() for a compression which requires a dictionary.
*
* Use ZBUFF_compressContinue() repetitively to consume input stream.
* *srcSizePtr and *dstCapacityPtr can be any size.
* The function will report how many bytes were read or written within *srcSizePtr and *dstCapacityPtr.
ext/zstd/lib/deprecated/zbuff.h view on Meta::CPAN
ZBUFF_DEPRECATED("use ZSTD_initDStream") size_t ZBUFF_decompressInit(ZBUFF_DCtx* dctx);
ZBUFF_DEPRECATED("use ZSTD_initDStream_usingDict") size_t ZBUFF_decompressInitDictionary(ZBUFF_DCtx* dctx, const void* dict, size_t dictSize);
ZBUFF_DEPRECATED("use ZSTD_decompressStream") size_t ZBUFF_decompressContinue(ZBUFF_DCtx* dctx,
void* dst, size_t* dstCapacityPtr,
const void* src, size_t* srcSizePtr);
/*-***************************************************************************
* Streaming decompression howto
*
* A ZBUFF_DCtx object is required to track streaming operations.
* Use ZBUFF_createDCtx() and ZBUFF_freeDCtx() to create/release resources.
* Use ZBUFF_decompressInit() to start a new decompression operation,
* or ZBUFF_decompressInitDictionary() if decompression requires a dictionary.
* Note that ZBUFF_DCtx objects can be re-init multiple times.
*
* Use ZBUFF_decompressContinue() repetitively to consume your input.
* *srcSizePtr and *dstCapacityPtr can be any size.
* The function will report how many bytes were read or written by modifying *srcSizePtr and *dstCapacityPtr.
* Note that it may not consume the entire input, in which case it's up to the caller to present remaining input again.
* The content of `dst` will be overwritten (up to *dstCapacityPtr) at each function call, so save its content if it matters, or change `dst`.
ext/zstd/lib/deprecated/zbuff_compress.c view on Meta::CPAN
/* *************************************
* Dependencies
***************************************/
#define ZBUFF_STATIC_LINKING_ONLY
#include "zbuff.h"
/*-***********************************************************
* Streaming compression
*
* A ZBUFF_CCtx object is required to track streaming operation.
* Use ZBUFF_createCCtx() and ZBUFF_freeCCtx() to create/release resources.
* Use ZBUFF_compressInit() to start a new compression operation.
* ZBUFF_CCtx objects can be reused multiple times.
*
* Use ZBUFF_compressContinue() repetitively to consume your input.
* *srcSizePtr and *dstCapacityPtr can be any size.
* The function will report how many bytes were read or written by modifying *srcSizePtr and *dstCapacityPtr.
* Note that it may not consume the entire input, in which case it's up to the caller to call again the function with remaining input.
* The content of dst will be overwritten (up to *dstCapacityPtr) at each function call, so save its content if it matters or change dst .
* @return : a hint to preferred nb of bytes to use as input for next function call (it's only a hint, to improve latency)
( run in 0.936 second using v1.01-cache-2.11-cpan-4d50c553e7e )