Atomic-Pipe
view release on metacpan or search on metacpan
Caveat: raw zstd dictionaries do not embed a dict-ID. As a result a
mismatched peer dictionary will silently decode to garbage rather than
fail. (Hard frame corruption -- truncated or invalid frames -- still
raises fatally.) Both ends must agree on byte-identical dictionary
content.
Performance
Compression is not just a wire-size optimization for Atomic::Pipe: when
messages exceed PIPE_BUF (typically 4096 bytes on Linux) the writer
must fragment them into multiple non-atomic chunks, and the reader must
reassemble them. Compressing the payload first frequently collapses a
multi-part message back into a single atomic burst, which avoids that
per-message protocol overhead entirely. As a result, on workloads
dominated by larger-than-PIPE_BUF messages, compression is often much
faster end-to-end than no compression, even after accounting for the
CPU cost of compress/decompress.
The kernel pipe buffer size (see "resize") does not affect this --
fragmentation is keyed on the POSIX PIPE_BUF atomic-write threshold,
not on the buffer capacity.
Benchmark: streaming JSON objects
Numbers below are from bench/zstd_compression.pl in the distribution.
The workload is a synthetic but representative stream of JSON log/event
objects sent in mixed-data mode via write_message. The corpus is
generated once and reused across all runs; sizes are JSON-encoded byte
counts.
L22 14.43 4.84 2.07x 51.6%
For this size distribution, levels 1..7 are all faster than no
compression (pipe back-pressure on the uncompressed run still
dominates).
Larger JSON (100 MB total, 20407 objects)
Object sizes 187 .. 10000 bytes, average ~5.1 KB, evenly distributed
across the 1..10 KB range. Most objects exceed PIPE_BUF, so the
uncompressed path pays the multi-part fragmentation cost on nearly
every message.
level raw MB/s wire MB ratio saved
plain 0.29 100.00 - -
L-3 287.85 35.61 2.81x 64.4%
L-1 273.56 33.92 2.95x 66.1%
L1 237.04 30.56 3.27x 69.4%
L3 (def) 207.61 30.25 3.31x 69.7%
L5 113.02 30.01 3.33x 70.0%
L9 39.35 29.93 3.34x 70.1%
L18 7.81 28.14 3.55x 71.9%
L22 7.85 28.14 3.55x 71.9%
Here the uncompressed run collapses to ~0.29 MB/s, while even modest
compression levels achieve 200+ MB/s -- a ~1000x throughput
improvement driven almost entirely by avoided fragmentation. Levels
above ~5 trade significant CPU for negligible additional ratio.
Pipe buffer size has minimal impact
The same 100 MB corpus, holding mode constant and varying the kernel
pipe buffer (32 KB, 128 KB, 512 KB, 1 MB), shows almost no movement
in either direction. The bottleneck is PIPE_BUF-aligned framing, not
buffer fill, so calling "resize" with a larger size will not rescue
an uncompressed large-message workload.
**Caveat:** raw zstd dictionaries do not embed a dict-ID. As a result a
**mismatched** peer dictionary will silently decode to garbage rather than
fail. (Hard frame corruption -- truncated or invalid frames -- still raises
fatally.) Both ends must agree on byte-identical dictionary content.
## Performance
Compression is not just a wire-size optimization for `Atomic::Pipe`: when
messages exceed `PIPE_BUF` (typically 4096 bytes on Linux) the writer must
fragment them into multiple non-atomic chunks, and the reader must reassemble
them. Compressing the payload first frequently collapses a multi-part message
back into a single atomic burst, which avoids that per-message protocol
overhead entirely. As a result, on workloads dominated by larger-than-PIPE\_BUF
messages, compression is often **much faster end-to-end than no compression**,
even after accounting for the CPU cost of compress/decompress.
The kernel pipe buffer size (see ["resize"](#resize)) does **not** affect this --
fragmentation is keyed on the POSIX `PIPE_BUF` atomic-write threshold, not on
the buffer capacity.
### Benchmark: streaming JSON objects
Numbers below are from `bench/zstd_compression.pl` in the distribution. The
workload is a synthetic but representative stream of JSON log/event objects
sent in mixed-data mode via `write_message`. The corpus is generated once and
reused across all runs; sizes are JSON-encoded byte counts.
Two corpora were measured:
L12 27.02 4.85 2.06x 51.5%
L22 14.43 4.84 2.07x 51.6%
For this size distribution, levels 1..7 are all faster than no compression
(pipe back-pressure on the uncompressed run still dominates).
- Larger JSON (100 MB total, 20407 objects)
Object sizes 187 .. 10000 bytes, average ~5.1 KB, evenly distributed across
the 1..10 KB range. Most objects exceed `PIPE_BUF`, so the uncompressed path
pays the multi-part fragmentation cost on nearly every message.
level raw MB/s wire MB ratio saved
plain 0.29 100.00 - -
L-3 287.85 35.61 2.81x 64.4%
L-1 273.56 33.92 2.95x 66.1%
L1 237.04 30.56 3.27x 69.4%
L3 (def) 207.61 30.25 3.31x 69.7%
L5 113.02 30.01 3.33x 70.0%
L9 39.35 29.93 3.34x 70.1%
L18 7.81 28.14 3.55x 71.9%
L22 7.85 28.14 3.55x 71.9%
Here the uncompressed run collapses to ~0.29 MB/s, while even modest
compression levels achieve 200+ MB/s -- a ~1000x throughput improvement
driven almost entirely by avoided fragmentation. Levels above ~5 trade
significant CPU for negligible additional ratio.
- Pipe buffer size has minimal impact
The same 100 MB corpus, holding mode constant and varying the kernel pipe
buffer (32 KB, 128 KB, 512 KB, 1 MB), shows almost no movement in either
direction. The bottleneck is `PIPE_BUF`-aligned framing, not buffer fill, so
calling ["resize"](#resize) with a larger size will not rescue an uncompressed
large-message workload.
lib/Atomic/Pipe.pm view on Meta::CPAN
B<Caveat:> raw zstd dictionaries do not embed a dict-ID. As a result a
B<mismatched> peer dictionary will silently decode to garbage rather than
fail. (Hard frame corruption -- truncated or invalid frames -- still raises
fatally.) Both ends must agree on byte-identical dictionary content.
=head2 Performance
Compression is not just a wire-size optimization for C<Atomic::Pipe>: when
messages exceed C<PIPE_BUF> (typically 4096 bytes on Linux) the writer must
fragment them into multiple non-atomic chunks, and the reader must reassemble
them. Compressing the payload first frequently collapses a multi-part message
back into a single atomic burst, which avoids that per-message protocol
overhead entirely. As a result, on workloads dominated by larger-than-PIPE_BUF
messages, compression is often B<much faster end-to-end than no compression>,
even after accounting for the CPU cost of compress/decompress.
The kernel pipe buffer size (see L</resize>) does B<not> affect this --
fragmentation is keyed on the POSIX C<PIPE_BUF> atomic-write threshold, not on
the buffer capacity.
=head3 Benchmark: streaming JSON objects
Numbers below are from C<bench/zstd_compression.pl> in the distribution. The
workload is a synthetic but representative stream of JSON log/event objects
sent in mixed-data mode via C<write_message>. The corpus is generated once and
reused across all runs; sizes are JSON-encoded byte counts.
Two corpora were measured:
lib/Atomic/Pipe.pm view on Meta::CPAN
L12 27.02 4.85 2.06x 51.5%
L22 14.43 4.84 2.07x 51.6%
For this size distribution, levels 1..7 are all faster than no compression
(pipe back-pressure on the uncompressed run still dominates).
=item Larger JSON (100 MB total, 20407 objects)
Object sizes 187 .. 10000 bytes, average ~5.1 KB, evenly distributed across
the 1..10 KB range. Most objects exceed C<PIPE_BUF>, so the uncompressed path
pays the multi-part fragmentation cost on nearly every message.
level raw MB/s wire MB ratio saved
plain 0.29 100.00 - -
L-3 287.85 35.61 2.81x 64.4%
L-1 273.56 33.92 2.95x 66.1%
L1 237.04 30.56 3.27x 69.4%
L3 (def) 207.61 30.25 3.31x 69.7%
L5 113.02 30.01 3.33x 70.0%
L9 39.35 29.93 3.34x 70.1%
L18 7.81 28.14 3.55x 71.9%
L22 7.85 28.14 3.55x 71.9%
Here the uncompressed run collapses to ~0.29 MB/s, while even modest
compression levels achieve 200+ MB/s -- a ~1000x throughput improvement
driven almost entirely by avoided fragmentation. Levels above ~5 trade
significant CPU for negligible additional ratio.
=item Pipe buffer size has minimal impact
The same 100 MB corpus, holding mode constant and varying the kernel pipe
buffer (32 KB, 128 KB, 512 KB, 1 MB), shows almost no movement in either
direction. The bottleneck is C<PIPE_BUF>-aligned framing, not buffer fill, so
calling L</resize> with a larger size will not rescue an uncompressed
large-message workload.
t/mixed_data_mode.t view on Meta::CPAN
is(
shift @got,
[line => "A Line\n"],
"Got the first line"
);
is(
shift @got,
[burst => "Interrupting cow!\n\n\n"],
"Got the burst between line fragments"
);
is(
shift @got,
[line => "Line start ...... line end\n"],
"Got the interrupted line"
);
ok(!$r->eof, "Not EOF");
t/mixed_data_mode.t view on Meta::CPAN
is(
shift @got,
[line => "A Line\n"],
"Got the first line"
);
is(
shift @got,
[message => ("aa" x PIPE_BUF)],
"Got the message between line fragments"
);
is(
shift @got,
[line => "Line start ...... line end\n"],
"Got the interrupted line"
);
ok(!$r->eof, "Not EOF");
( run in 0.963 second using v1.01-cache-2.11-cpan-364913b4093 )