view release on metacpan or  search on metacpan

README.md  view on Meta::CPAN

sent in mixed-data mode via `write_message`. The corpus is generated once and
reused across all runs; sizes are JSON-encoded byte counts.

Two corpora were measured:

- Small JSON (10 MB total, 11785 objects)

    Object sizes 181 .. 1977 bytes, average ~890 B; ~37% of objects under 500 B.
    Most messages fit in a single `PIPE_BUF` burst regardless of compression.

        level     raw MB/s   wire MB    ratio   saved
        plain         9.74    10.00       -        -
        L-3          15.98     6.68    1.50x    33.2%
        L1           24.55     4.92    2.03x    50.8%
        L3 (def)     27.79     4.91    2.04x    50.9%
        L5           46.34     4.87    2.05x    51.3%
        L7           63.72     4.87    2.05x    51.3%
        L12          27.02     4.85    2.06x    51.5%
        L22          14.43     4.84    2.07x    51.6%

    For this size distribution, levels 1..7 are all faster than no compression
    (pipe back-pressure on the uncompressed run still dominates).

- Larger JSON (100 MB total, 20407 objects)

    Object sizes 187 .. 10000 bytes, average ~5.1 KB, evenly distributed across
    the 1..10 KB range. Most objects exceed `PIPE_BUF`, so the uncompressed path
    pays the multi-part fragmentation cost on nearly every message.

        level     raw MB/s   wire MB    ratio   saved
        plain         0.29   100.00       -        -
        L-3         287.85    35.61    2.81x    64.4%
        L-1         273.56    33.92    2.95x    66.1%
        L1          237.04    30.56    3.27x    69.4%
        L3 (def)    207.61    30.25    3.31x    69.7%
        L5          113.02    30.01    3.33x    70.0%
        L9           39.35    29.93    3.34x    70.1%
        L18           7.81    28.14    3.55x    71.9%
        L22           7.85    28.14    3.55x    71.9%

    Here the uncompressed run collapses to ~0.29 MB/s, while even modest
    compression levels achieve 200+ MB/s -- a ~1000x throughput improvement
    driven almost entirely by avoided fragmentation. Levels above ~5 trade
    significant CPU for negligible additional ratio.

- Pipe buffer size has minimal impact

    The same 100 MB corpus, holding mode constant and varying the kernel pipe
    buffer (32 KB, 128 KB, 512 KB, 1 MB), shows almost no movement in either
    direction. The bottleneck is `PIPE_BUF`-aligned framing, not buffer fill, so
    calling ["resize"](#resize) with a larger size will not rescue an uncompressed
    large-message workload.

### Practical guidance

- If your messages are routinely larger than `PIPE_BUF` (~4 KB), enabling
compression is almost always a throughput win, not just a bandwidth win.
- For mixed JSON-like payloads, **level 1** or the default **level 3** are good
starting points. Level -3 is the throughput champion when CPU is precious and
some ratio can be sacrificed.
- Levels above ~7 buy single-digit-percent ratio gains for multi-x CPU cost; in
an IPC path they are rarely worth it.
- A custom dictionary (["Custom dictionary"](#custom-dictionary)) helps most when payloads are
small and share structure -- e.g. identical JSON keys across every message.

These results depend heavily on payload entropy and CPU. Re-run
`bench/zstd_compression.pl` against a representative slice of your own data
before committing to a level.

# METHODS

## CLASS METHODS

- $bytes = Atomic::Pipe->PIPE\_BUF

    Get the maximum number of bytes for an atomic write to a pipe.

- $bool = Atomic::Pipe->HAVE\_IO\_SELECT

    True if [IO::Select](https://metacpan.org/pod/IO%3A%3ASelect) is available on this system. When available, it is used by
    default in `fill_buffer()` to efficiently wait for pipe readability instead of
    relying on blocking `sysread()` with an EINTR retry loop.

- ($r, $w) = Atomic::Pipe->pair
- ($r, $w) = Atomic::Pipe->pair(%params)

    Create a pipe, returns a list consisting of a reader and a writer.

    All constructors accept the same optional `%params`: the compression options
    documented in ["COMPRESSION"](#compression), and `mixed_data_mode => 1` (see
    ["MIXED DATA MODE"](#mixed-data-mode)).

- $p = Atomic::Pipe->new
- $p = Atomic::Pipe->new(%params)

    If you really must have a `new()` method it is here for you to abuse. The
    returned pipe has both handles, it is your job to then turn it into 2 clones
    one with the reader and one with the writer. It is also your job to make sure
    you do not have too many handles floating around preventing an EOF.

- $r = Atomic::Pipe->read\_fifo($FIFO\_PATH, %params)
- $w = Atomic::Pipe->write\_fifo($FIFO\_PATH, %params)

    These 2 constructors let you connect to a FIFO by filesystem path.

    The interface difference (read\_fifo and write\_fifo vs specifying a mode) is
    because the modes to use for fifo's are not obvious (`'+<'` for reading).

    **NOTE:** THERE IS NO EOF for the read-end in the process that created the fifo.
    You need to figure out when the last message is received on your own somehow.
    If you use blocking reads in a loop with no loop exit condition then the loop
    will never end even after all writers are gone.

- $p = Atomic::Pipe->from\_fh($fh, %params)
- $p = Atomic::Pipe->from\_fh($mode, $fh, %params)

    Create an instance around an existing filehandle (A clone of the handle will be
    made and kept internally).

    This will fail if the handle is not a pipe.