Compress-Stream-Zstd
view release on metacpan or search on metacpan
ext/zstd/doc/zstd_manual.html view on Meta::CPAN
note : since v1.3.0, ZSTD_CStream and ZSTD_CCtx are the same thing.
Parameters are sticky : when starting a new compression on the same context,
it will re-use the same sticky parameters as previous compression session.
When in doubt, it's recommended to fully initialize the context before usage.
Use ZSTD_CCtx_reset() to reset the context and ZSTD_CCtx_setParameter(),
ZSTD_CCtx_setPledgedSrcSize(), or ZSTD_CCtx_loadDictionary() and friends to
set more specific parameters, the pledged source size, or load a dictionary.
Use ZSTD_compressStream2() with ZSTD_e_continue as many times as necessary to
consume input stream. The function will automatically update both `pos`
fields within `input` and `output`.
Note that the function may not consume the entire input, for example, because
the output buffer is already full, in which case `input.pos < input.size`.
The caller must check if input has been entirely consumed.
If not, the caller must make some room to receive more compressed data,
and then present again remaining input data.
note: ZSTD_e_continue is guaranteed to make some forward progress when called,
but doesn't guarantee maximal forward progress. This is especially relevant
when compressing with multiple threads. The call won't block if it can
consume some input, but if it can't it will wait for some, but not all,
output to be flushed.
@return : provides a minimum amount of data remaining to be flushed from internal buffers
or an error code, which can be tested using ZSTD_isError().
At any moment, it's possible to flush whatever data might remain stuck within internal buffer,
using ZSTD_compressStream2() with ZSTD_e_flush. `output->pos` will be updated.
Note that, if `output->size` is too small, a single invocation with ZSTD_e_flush might not be enough (return code > 0).
In which case, make some room to receive more compressed data, and call again ZSTD_compressStream2() with ZSTD_e_flush.
You must continue calling ZSTD_compressStream2() with ZSTD_e_flush until it returns 0, at which point you can change the
operation.
note: ZSTD_e_flush will flush as much output as possible, meaning when compressing with multiple threads, it will
block until the flush is complete or the output buffer is full.
@return : 0 if internal buffers are entirely flushed,
>0 if some data still present within internal buffer (the value is minimal estimation of remaining size),
or an error code, which can be tested using ZSTD_isError().
Calling ZSTD_compressStream2() with ZSTD_e_end instructs to finish a frame.
It will perform a flush and write frame epilogue.
The epilogue is required for decoders to consider a frame completed.
flush operation is the same, and follows same rules as calling ZSTD_compressStream2() with ZSTD_e_flush.
You must continue calling ZSTD_compressStream2() with ZSTD_e_end until it returns 0, at which point you are free to
start a new frame.
note: ZSTD_e_end will flush as much output as possible, meaning when compressing with multiple threads, it will
block until the flush is complete or the output buffer is full.
@return : 0 if frame fully completed and fully flushed,
>0 if some data still present within internal buffer (the value is minimal estimation of remaining size),
or an error code, which can be tested using ZSTD_isError().
<BR></pre>
<pre><b>typedef ZSTD_CCtx ZSTD_CStream; </b>/**< CCtx and CStream are now effectively same object (>= v1.3.0) */<b>
</b></pre><BR>
<h3>ZSTD_CStream management functions</h3><pre></pre><b><pre>ZSTD_CStream* ZSTD_createCStream(void);
size_t ZSTD_freeCStream(ZSTD_CStream* zcs); </b>/* accept NULL pointer */<b>
</pre></b><BR>
<h3>Streaming compression functions</h3><pre></pre><b><pre>typedef enum {
ZSTD_e_continue=0, </b>/* collect more data, encoder decides when to output compressed result, for optimal compression ratio */<b>
ZSTD_e_flush=1, </b>/* flush any data provided so far,<b>
* it creates (at least) one new block, that can be decoded immediately on reception;
* frame will continue: any future data can still reference previously compressed data, improving compression.
* note : multithreaded compression will block to flush as much output as possible. */
ZSTD_e_end=2 </b>/* flush any remaining data _and_ close current frame.<b>
* note that frame is only closed after compressed data is fully flushed (return value == 0).
* After that point, any additional data starts a new frame.
* note : each frame is independent (does not reference any content from previous frame).
: note : multithreaded compression will block to flush as much output as possible. */
} ZSTD_EndDirective;
</pre></b><BR>
<pre><b>size_t ZSTD_compressStream2( ZSTD_CCtx* cctx,
ZSTD_outBuffer* output,
ZSTD_inBuffer* input,
ZSTD_EndDirective endOp);
</b><p> Behaves about the same as ZSTD_compressStream, with additional control on end directive.
- Compression parameters are pushed into CCtx before starting compression, using ZSTD_CCtx_set*()
- Compression parameters cannot be changed once compression is started (save a list of exceptions in multi-threading mode)
- output->pos must be <= dstCapacity, input->pos must be <= srcSize
- output->pos and input->pos will be updated. They are guaranteed to remain below their respective limit.
- endOp must be a valid directive
- When nbWorkers==0 (default), function is blocking : it completes its job before returning to caller.
- When nbWorkers>=1, function is non-blocking : it copies a portion of input, distributes jobs to internal worker threads, flush to output whatever is available,
and then immediately returns, just indicating that there is some data remaining to be flushed.
The function nonetheless guarantees forward progress : it will return only after it reads or write at least 1+ byte.
- Exception : if the first call requests a ZSTD_e_end directive and provides enough dstCapacity, the function delegates to ZSTD_compress2() which is always blocking.
- @return provides a minimum amount of data remaining to be flushed from internal buffers
or an error code, which can be tested using ZSTD_isError().
if @return != 0, flush is not fully completed, there is still some data left within internal buffers.
This is useful for ZSTD_e_flush, since in this case more flushes are necessary to empty all buffers.
For ZSTD_e_end, @return == 0 when internal buffers are fully flushed and frame is completed.
- after a ZSTD_e_end directive, if internal buffer is not fully flushed (@return != 0),
only ZSTD_e_end or ZSTD_e_flush operations are allowed.
Before starting a new compression job, or changing compression parameters,
it is required to fully flush internal buffers.
</p></pre><BR>
<pre><b>size_t ZSTD_CStreamInSize(void); </b>/**< recommended size for input buffer */<b>
</b></pre><BR>
<pre><b>size_t ZSTD_CStreamOutSize(void); </b>/**< recommended size for output buffer. Guarantee to successfully flush at least one complete compressed block. */<b>
</b></pre><BR>
<pre><b>size_t ZSTD_initCStream(ZSTD_CStream* zcs, int compressionLevel);
</b>/*!<b>
* Alternative for ZSTD_compressStream2(zcs, output, input, ZSTD_e_continue).
* NOTE: The return value is different. ZSTD_compressStream() returns a hint for
* the next read size (if non-zero and not an error). ZSTD_compressStream2()
* returns the minimum nb of bytes left to flush (if non-zero and not an error).
*/
size_t ZSTD_compressStream(ZSTD_CStream* zcs, ZSTD_outBuffer* output, ZSTD_inBuffer* input);
</b>/*! Equivalent to ZSTD_compressStream2(zcs, output, &emptyInput, ZSTD_e_flush). */<b>
size_t ZSTD_flushStream(ZSTD_CStream* zcs, ZSTD_outBuffer* output);
</b>/*! Equivalent to ZSTD_compressStream2(zcs, output, &emptyInput, ZSTD_e_end). */<b>
size_t ZSTD_endStream(ZSTD_CStream* zcs, ZSTD_outBuffer* output);
</b><p>
ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only);
ZSTD_CCtx_refCDict(zcs, NULL); // clear the dictionary (if any)
ZSTD_CCtx_setParameter(zcs, ZSTD_c_compressionLevel, compressionLevel);
Note that ZSTD_initCStream() clears any previously set dictionary. Use the new API
to compress with a dictionary.
</p></pre><BR>
<a name="Chapter9"></a><h2>Streaming decompression - HowTo</h2><pre>
A ZSTD_DStream object is required to track streaming operations.
Use ZSTD_createDStream() and ZSTD_freeDStream() to create/release resources.
ZSTD_DStream objects can be re-used multiple times.
Use ZSTD_initDStream() to start a new decompression operation.
@return : recommended first input size
Alternatively, use advanced API to set specific properties.
Use ZSTD_decompressStream() repetitively to consume your input.
The function will update both `pos` fields.
If `input.pos < input.size`, some input has not been consumed.
It's up to the caller to present again remaining data.
The function tries to flush all data decoded immediately, respecting output buffer size.
If `output.pos < output.size`, decoder has flushed everything it could.
But if `output.pos == output.size`, there might be some data left within internal buffers.,
In which case, call ZSTD_decompressStream() again to flush whatever remains in the buffer.
Note : with no additional input provided, amount of data flushed is necessarily <= ZSTD_BLOCKSIZE_MAX.
@return : 0 when a frame is completely decoded and fully flushed,
or an error code, which can be tested using ZSTD_isError(),
or any other value > 0, which means there is still some decoding or flushing to do to complete current frame :
the return value is a suggested next input size (just a hint for better latency)
that will never request more than the remaining frame size.
<BR></pre>
<pre><b>typedef ZSTD_DCtx ZSTD_DStream; </b>/**< DCtx and DStream are now effectively same object (>= v1.3.0) */<b>
</b></pre><BR>
<h3>ZSTD_DStream management functions</h3><pre></pre><b><pre>ZSTD_DStream* ZSTD_createDStream(void);
size_t ZSTD_freeDStream(ZSTD_DStream* zds); </b>/* accept NULL pointer */<b>
</pre></b><BR>
<h3>Streaming decompression functions</h3><pre></pre><b><pre></pre></b><BR>
<pre><b>size_t ZSTD_initDStream(ZSTD_DStream* zds);
</b><p> Initialize/reset DStream state for new decompression operation.
Call before new decompression operation using same DStream.
Note : This function is redundant with the advanced API and equivalent to:
ZSTD_DCtx_reset(zds, ZSTD_reset_session_only);
ZSTD_DCtx_refDDict(zds, NULL);
</p></pre><BR>
<pre><b>size_t ZSTD_decompressStream(ZSTD_DStream* zds, ZSTD_outBuffer* output, ZSTD_inBuffer* input);
</b><p> Streaming decompression function.
Call repetitively to consume full input updating it as necessary.
Function will update both input and output `pos` fields exposing current state via these fields:
- `input.pos < input.size`, some input remaining and caller should provide remaining input
on the next call.
- `output.pos < output.size`, decoder finished and flushed all remaining buffers.
- `output.pos == output.size`, potentially uncflushed data present in the internal buffers,
call ZSTD_decompressStream() again to flush remaining data to output.
Note : with no additional input, amount of data flushed <= ZSTD_BLOCKSIZE_MAX.
@return : 0 when a frame is completely decoded and fully flushed,
or an error code, which can be tested using ZSTD_isError(),
or any other value > 0, which means there is some decoding or flushing to do to complete current frame.
</p></pre><BR>
<pre><b>size_t ZSTD_DStreamInSize(void); </b>/*!< recommended size for input buffer */<b>
</b></pre><BR>
<pre><b>size_t ZSTD_DStreamOutSize(void); </b>/*!< recommended size for output buffer. Guarantee to successfully flush at least one complete block in all circumstances. */<b>
</b></pre><BR>
<a name="Chapter10"></a><h2>Simple dictionary API</h2><pre></pre>
<pre><b>size_t ZSTD_compress_usingDict(ZSTD_CCtx* ctx,
void* dst, size_t dstCapacity,
const void* src, size_t srcSize,
const void* dict,size_t dictSize,
int compressionLevel);
</b><p> Compression at an explicit compression level using a Dictionary.
A dictionary can be any arbitrary data segment (also called a prefix),
or a buffer with specified information (see zdict.h).
Note : This function loads the dictionary, resulting in significant startup delay.
It's intended for a dictionary used only once.
Note 2 : When `dict == NULL || dictSize < 8` no dictionary is used.
</p></pre><BR>
<pre><b>size_t ZSTD_decompress_usingDict(ZSTD_DCtx* dctx,
void* dst, size_t dstCapacity,
const void* src, size_t srcSize,
const void* dict,size_t dictSize);
</b><p> Decompression using a known Dictionary.
Dictionary must be identical to the one used during compression.
Note : This function loads the dictionary, resulting in significant startup delay.
It's intended for a dictionary used only once.
Note : When `dict == NULL || dictSize < 8` no dictionary is used.
</p></pre><BR>
<a name="Chapter11"></a><h2>Bulk processing dictionary API</h2><pre></pre>
<pre><b>ZSTD_CDict* ZSTD_createCDict(const void* dictBuffer, size_t dictSize,
int compressionLevel);
</b><p> When compressing multiple messages or blocks using the same dictionary,
it's recommended to digest the dictionary only once, since it's a costly operation.
ZSTD_createCDict() will create a state from digesting a dictionary.
The resulting state can be used for future compression operations with very limited startup cost.
ZSTD_CDict can be created once and shared by multiple threads concurrently, since its usage is read-only.
@dictBuffer can be released after ZSTD_CDict creation, because its content is copied within CDict.
Note 1 : Consider experimental function `ZSTD_createCDict_byReference()` if you prefer to not duplicate @dictBuffer content.
Note 2 : A ZSTD_CDict can be created from an empty @dictBuffer,
in which case the only thing that it transports is the @compressionLevel.
This can be useful in a pipeline featuring ZSTD_compress_usingCDict() exclusively,
expecting a ZSTD_CDict parameter with any data, including those without a known dictionary.
</p></pre><BR>
<pre><b>size_t ZSTD_freeCDict(ZSTD_CDict* CDict);
</b><p> Function frees memory allocated by ZSTD_createCDict().
If a NULL pointer is passed, no operation is performed.
</p></pre><BR>
<pre><b>size_t ZSTD_compress_usingCDict(ZSTD_CCtx* cctx,
void* dst, size_t dstCapacity,
const void* src, size_t srcSize,
const ZSTD_CDict* cdict);
</b><p> Compression using a digested Dictionary.
Recommended when same dictionary is used multiple times.
Note : compression level is _decided at dictionary creation time_,
and frame parameters are hardcoded (dictID=yes, contentSize=yes, checksum=no)
</p></pre><BR>
<pre><b>ZSTD_DDict* ZSTD_createDDict(const void* dictBuffer, size_t dictSize);
</b><p> Create a digested dictionary, ready to start decompression operation without startup delay.
dictBuffer can be released after DDict creation, as its content is copied inside DDict.
</p></pre><BR>
<pre><b>size_t ZSTD_freeDDict(ZSTD_DDict* ddict);
</b><p> Function frees memory allocated with ZSTD_createDDict()
If a NULL pointer is passed, no operation is performed.
</p></pre><BR>
<pre><b>size_t ZSTD_decompress_usingDDict(ZSTD_DCtx* dctx,
void* dst, size_t dstCapacity,
const void* src, size_t srcSize,
const ZSTD_DDict* ddict);
</b><p> Decompression using a digested Dictionary.
Recommended when same dictionary is used multiple times.
</p></pre><BR>
<a name="Chapter12"></a><h2>Dictionary helper functions</h2><pre></pre>
<pre><b>unsigned ZSTD_getDictID_fromDict(const void* dict, size_t dictSize);
</b><p> Provides the dictID stored within dictionary.
if @return == 0, the dictionary is not conformant with Zstandard specification.
It can still be loaded, but as a content-only dictionary.
</p></pre><BR>
<pre><b>unsigned ZSTD_getDictID_fromCDict(const ZSTD_CDict* cdict);
</b><p> Provides the dictID of the dictionary loaded into `cdict`.
If @return == 0, the dictionary is not conformant to Zstandard specification, or empty.
Non-conformant dictionaries can still be loaded, but as content-only dictionaries.
</p></pre><BR>
<pre><b>unsigned ZSTD_getDictID_fromDDict(const ZSTD_DDict* ddict);
</b><p> Provides the dictID of the dictionary loaded into `ddict`.
If @return == 0, the dictionary is not conformant to Zstandard specification, or empty.
Non-conformant dictionaries can still be loaded, but as content-only dictionaries.
</p></pre><BR>
<pre><b>unsigned ZSTD_getDictID_fromFrame(const void* src, size_t srcSize);
</b><p> Provides the dictID required to decompressed the frame stored within `src`.
If @return == 0, the dictID could not be decoded.
This could for one of the following reasons :
- The frame does not require a dictionary to be decoded (most common case).
- The frame was built with dictID intentionally removed. Whatever dictionary is necessary is a hidden piece of information.
Note : this use case also happens when using a non-conformant dictionary.
- `srcSize` is too small, and as a result, the frame header could not be decoded (only possible if `srcSize < ZSTD_FRAMEHEADERSIZE_MAX`).
- This is not a Zstandard frame.
When identifying the exact failure cause, it's possible to use ZSTD_getFrameHeader(), which will provide a more precise error code.
</p></pre><BR>
<a name="Chapter13"></a><h2>Advanced dictionary and prefix API (Requires v1.4.0+)</h2><pre>
This API allows dictionaries to be used with ZSTD_compress2(),
ZSTD_compressStream2(), and ZSTD_decompressDCtx().
Dictionaries are sticky, they remain valid when same context is re-used,
they only reset when the context is reset
with ZSTD_reset_parameters or ZSTD_reset_session_and_parameters.
In contrast, Prefixes are single-use.
<BR></pre>
<pre><b>size_t ZSTD_CCtx_loadDictionary(ZSTD_CCtx* cctx, const void* dict, size_t dictSize);
</b><p> Create an internal CDict from `dict` buffer.
Decompression will have to use same dictionary.
@result : 0, or an error code (which can be tested with ZSTD_isError()).
Special: Loading a NULL (or 0-size) dictionary invalidates previous dictionary,
meaning "return to no-dictionary mode".
Note 1 : Dictionary is sticky, it will be used for all future compressed frames,
until parameters are reset, a new dictionary is loaded, or the dictionary
is explicitly invalidated by loading a NULL dictionary.
Note 2 : Loading a dictionary involves building tables.
It's also a CPU consuming operation, with non-negligible impact on latency.
Tables are dependent on compression parameters, and for this reason,
compression parameters can no longer be changed after loading a dictionary.
Note 3 :`dict` content will be copied internally.
Use experimental ZSTD_CCtx_loadDictionary_byReference() to reference content instead.
In such a case, dictionary buffer must outlive its users.
Note 4 : Use ZSTD_CCtx_loadDictionary_advanced()
to precisely select how dictionary content must be interpreted.
Note 5 : This method does not benefit from LDM (long distance mode).
If you want to employ LDM on some large dictionary content,
prefer employing ZSTD_CCtx_refPrefix() described below.
</p></pre><BR>
<pre><b>size_t ZSTD_CCtx_refCDict(ZSTD_CCtx* cctx, const ZSTD_CDict* cdict);
</b><p> Reference a prepared dictionary, to be used for all future compressed frames.
Note that compression parameters are enforced from within CDict,
and supersede any compression parameter previously set within CCtx.
The parameters ignored are labelled as "superseded-by-cdict" in the ZSTD_cParameter enum docs.
The ignored parameters will be used again if the CCtx is returned to no-dictionary mode.
The dictionary will remain valid for future compressed frames using same CCtx.
@result : 0, or an error code (which can be tested with ZSTD_isError()).
Special : Referencing a NULL CDict means "return to no-dictionary mode".
Note 1 : Currently, only one dictionary can be managed.
Referencing a new dictionary effectively "discards" any previous one.
Note 2 : CDict is just referenced, its lifetime must outlive its usage within CCtx.
</p></pre><BR>
<pre><b>size_t ZSTD_CCtx_refPrefix(ZSTD_CCtx* cctx,
const void* prefix, size_t prefixSize);
</b><p> Reference a prefix (single-usage dictionary) for next compressed frame.
A prefix is **only used once**. Tables are discarded at end of frame (ZSTD_e_end).
Decompression will need same prefix to properly regenerate data.
Compressing with a prefix is similar in outcome as performing a diff and compressing it,
but performs much faster, especially during decompression (compression speed is tunable with compression level).
This method is compatible with LDM (long distance mode).
@result : 0, or an error code (which can be tested with ZSTD_isError()).
ext/zstd/doc/zstd_manual.html view on Meta::CPAN
- Caller must ensure there is enough space in `dst` to store compressed data under worst case scenario.
Worst case evaluation is provided by ZSTD_compressBound().
ZSTD_compressContinue() doesn't guarantee recover after a failed compression.
- ZSTD_compressContinue() presumes prior input ***is still accessible and unmodified*** (up to maximum distance size, see WindowLog).
It remembers all previous contiguous blocks, plus one separated memory segment (which can itself consists of multiple contiguous blocks)
- ZSTD_compressContinue() detects that prior input has been overwritten when `src` buffer overlaps.
In which case, it will "discard" the relevant memory section from its history.
Finish a frame with ZSTD_compressEnd(), which will write the last block(s) and optional checksum.
It's possible to use srcSize==0, in which case, it will write a final empty block to end the frame.
Without last block mark, frames are considered unfinished (hence corrupted) by compliant decoders.
`ZSTD_CCtx` object can be re-used (ZSTD_compressBegin()) to compress again.
<BR></pre>
<h3>Buffer-less streaming compression functions</h3><pre></pre><b><pre>ZSTD_DEPRECATED("The buffer-less API is deprecated in favor of the normal streaming API. See docs.")
ZSTDLIB_STATIC_API size_t ZSTD_compressBegin(ZSTD_CCtx* cctx, int compressionLevel);
ZSTD_DEPRECATED("The buffer-less API is deprecated in favor of the normal streaming API. See docs.")
ZSTDLIB_STATIC_API size_t ZSTD_compressBegin_usingDict(ZSTD_CCtx* cctx, const void* dict, size_t dictSize, int compressionLevel);
ZSTD_DEPRECATED("The buffer-less API is deprecated in favor of the normal streaming API. See docs.")
ZSTDLIB_STATIC_API size_t ZSTD_compressBegin_usingCDict(ZSTD_CCtx* cctx, const ZSTD_CDict* cdict); </b>/**< note: fails if cdict==NULL */<b>
</pre></b><BR>
<pre><b>size_t ZSTD_copyCCtx(ZSTD_CCtx* cctx, const ZSTD_CCtx* preparedCCtx, unsigned long long pledgedSrcSize); </b>/**< note: if pledgedSrcSize is not known, use ZSTD_CONTENTSIZE_UNKNOWN */<b>
</b></pre><BR>
<pre><b>size_t ZSTD_compressBegin_advanced(ZSTD_CCtx* cctx, const void* dict, size_t dictSize, ZSTD_parameters params, unsigned long long pledgedSrcSize); </b>/**< pledgedSrcSize : If srcSize is not known at init time, use ZSTD_CONTENTSIZE_UNKNOWN */...
</b></pre><BR>
<a name="Chapter22"></a><h2>Buffer-less streaming decompression (synchronous mode)</h2><pre>
A ZSTD_DCtx object is required to track streaming operations.
Use ZSTD_createDCtx() / ZSTD_freeDCtx() to manage it.
A ZSTD_DCtx object can be re-used multiple times.
First typical operation is to retrieve frame parameters, using ZSTD_getFrameHeader().
Frame header is extracted from the beginning of compressed frame, so providing only the frame's beginning is enough.
Data fragment must be large enough to ensure successful decoding.
`ZSTD_frameHeaderSize_max` bytes is guaranteed to always be large enough.
result : 0 : successful decoding, the `ZSTD_frameHeader` structure is correctly filled.
>0 : `srcSize` is too small, please provide at least result bytes on next attempt.
errorCode, which can be tested using ZSTD_isError().
It fills a ZSTD_frameHeader structure with important information to correctly decode the frame,
such as the dictionary ID, content size, or maximum back-reference distance (`windowSize`).
Note that these values could be wrong, either because of data corruption, or because a 3rd party deliberately spoofs false information.
As a consequence, check that values remain within valid application range.
For example, do not allocate memory blindly, check that `windowSize` is within expectation.
Each application can set its own limits, depending on local restrictions.
For extended interoperability, it is recommended to support `windowSize` of at least 8 MB.
ZSTD_decompressContinue() needs previous data blocks during decompression, up to `windowSize` bytes.
ZSTD_decompressContinue() is very sensitive to contiguity,
if 2 blocks don't follow each other, make sure that either the compressor breaks contiguity at the same place,
or that previous contiguous segment is large enough to properly handle maximum back-reference distance.
There are multiple ways to guarantee this condition.
The most memory efficient way is to use a round buffer of sufficient size.
Sufficient size is determined by invoking ZSTD_decodingBufferSize_min(),
which can return an error code if required value is too large for current system (in 32-bits mode).
In a round buffer methodology, ZSTD_decompressContinue() decompresses each block next to previous one,
up to the moment there is not enough room left in the buffer to guarantee decoding another full block,
which maximum size is provided in `ZSTD_frameHeader` structure, field `blockSizeMax`.
At which point, decoding can resume from the beginning of the buffer.
Note that already decoded data stored in the buffer should be flushed before being overwritten.
There are alternatives possible, for example using two or more buffers of size `windowSize` each, though they consume more memory.
Finally, if you control the compression process, you can also ignore all buffer size rules,
as long as the encoder and decoder progress in "lock-step",
aka use exactly the same buffer sizes, break contiguity at the same place, etc.
Once buffers are setup, start decompression, with ZSTD_decompressBegin().
If decompression requires a dictionary, use ZSTD_decompressBegin_usingDict() or ZSTD_decompressBegin_usingDDict().
Then use ZSTD_nextSrcSizeToDecompress() and ZSTD_decompressContinue() alternatively.
ZSTD_nextSrcSizeToDecompress() tells how many bytes to provide as 'srcSize' to ZSTD_decompressContinue().
ZSTD_decompressContinue() requires this _exact_ amount of bytes, or it will fail.
result of ZSTD_decompressContinue() is the number of bytes regenerated within 'dst' (necessarily <= dstCapacity).
It can be zero : it just means ZSTD_decompressContinue() has decoded some metadata item.
It can also be an error code, which can be tested with ZSTD_isError().
A frame is fully decoded when ZSTD_nextSrcSizeToDecompress() returns zero.
Context can then be reset to start a new decompression.
Note : it's possible to know if next input to present is a header or a block, using ZSTD_nextInputType().
This information is not required to properly decode a frame.
== Special case : skippable frames
Skippable frames allow integration of user-defined data into a flow of concatenated frames.
Skippable frames will be ignored (skipped) by decompressor.
The format of skippable frames is as follows :
a) Skippable frame ID - 4 Bytes, Little endian format, any value from 0x184D2A50 to 0x184D2A5F
b) Frame Size - 4 Bytes, Little endian format, unsigned 32-bits
c) Frame Content - any content (User Data) of length equal to Frame Size
For skippable frames ZSTD_getFrameHeader() returns zfhPtr->frameType==ZSTD_skippableFrame.
For skippable frames ZSTD_decompressContinue() always returns 0 : it only skips the content.
<BR></pre>
<h3>Buffer-less streaming decompression functions</h3><pre></pre><b><pre></pre></b><BR>
<pre><b>ZSTDLIB_STATIC_API size_t ZSTD_decodingBufferSize_min(unsigned long long windowSize, unsigned long long frameContentSize); </b>/**< when frame content size is not known, pass in frameContentSize == ZSTD_CONTENTSIZE_UNKNOWN */<b>
</b></pre><BR>
<pre><b>typedef enum { ZSTDnit_frameHeader, ZSTDnit_blockHeader, ZSTDnit_block, ZSTDnit_lastBlock, ZSTDnit_checksum, ZSTDnit_skippableFrame } ZSTD_nextInputType_e;
</b></pre><BR>
<a name="Chapter23"></a><h2>Block level API (DEPRECATED)</h2><pre></pre>
<pre><b></b><p> You can get the frame header down to 2 bytes by setting:
- ZSTD_c_format = ZSTD_f_zstd1_magicless
- ZSTD_c_contentSizeFlag = 0
- ZSTD_c_checksumFlag = 0
- ZSTD_c_dictIDFlag = 0
This API is not as well tested as our normal API, so we recommend not using it.
We will be removing it in a future version. If the normal API doesn't provide
the functionality you need, please open a GitHub issue.
Block functions produce and decode raw zstd blocks, without frame metadata.
Frame metadata cost is typically ~12 bytes, which can be non-negligible for very small blocks (< 100 bytes).
But users will have to take in charge needed metadata to regenerate data, such as compressed and content sizes.
A few rules to respect :
- Compressing and decompressing require a context structure
+ Use ZSTD_createCCtx() and ZSTD_createDCtx()
- It is necessary to init context before starting
+ compression : any ZSTD_compressBegin*() variant, including with dictionary
+ decompression : any ZSTD_decompressBegin*() variant, including with dictionary
- Block size is limited, it must be <= ZSTD_getBlockSize() <= ZSTD_BLOCKSIZE_MAX == 128 KB
+ If input is larger than a block size, it's necessary to split input data into multiple blocks
+ For inputs larger than a single block, consider using regular ZSTD_compress() instead.
Frame metadata is not that costly, and quickly becomes negligible as source size grows larger than a block.
- When a block is considered not compressible enough, ZSTD_compressBlock() result will be 0 (zero) !
===> In which case, nothing is produced into `dst` !
+ User __must__ test for such outcome and deal directly with uncompressed data
+ A block cannot be declared incompressible if ZSTD_compressBlock() return value was != 0.
Doing so would mess up with statistics history, leading to potential data corruption.
+ ZSTD_decompressBlock() _doesn't accept uncompressed data as input_ !!
+ In case of multiple successive blocks, should some of them be uncompressed,
decoder must be informed of their existence in order to follow proper history.
Use ZSTD_insertBlock() for such a case.
</p></pre><BR>
<h3>Raw zstd block functions</h3><pre></pre><b><pre>ZSTD_DEPRECATED("The block API is deprecated in favor of the normal compression API. See docs.")
( run in 1.346 second using v1.01-cache-2.11-cpan-39bf76dae61 )