view release on metacpan or  search on metacpan

lib/ClickHouse/Encoder.pm  view on Meta::CPAN

=item C<'zstd'>

ZSTD-compressed via L<Compress::Zstd>.

=item C<'auto'>

Try LZ4 first; if the result is C<E<gt>=> the input, fall back to
C<'none'>. Mirrors CH's own C<CompressedWriteBuffer> behavior for
incompressible payloads.

=item C<'none'>

No compression but still wrapped in the framing (method tag C<0x02>).
Useful when the wire context requires compressed-block framing but
the payload doesn't benefit from compression.

=back

The checksum is CityHash128 in the "cityhash102" variant
(ClickHouse's namespace fork of Google CityHash v1.0.2). This
module bundles a port of that algorithm in F<cityhash.c>, exposed
as the XSUB C<_cityhash128>; both C<compress_native_block> and
L</decompress_native_block> default to it. Pass an explicit
C<hasher =E<gt> $coderef> only if you want to plug in a different
implementation.

C<Compress::LZ4> is required for C<'lz4'> / C<'auto'> mode;
C<Compress::Zstd> for C<'zstd'>. Both are listed as runtime
C<recommends>.

=head2 decompress_native_block

    my ($plain, $consumed) = ClickHouse::Encoder->decompress_native_block(
        $framed);                          # default hasher = bundled
    my $plain = ClickHouse::Encoder->decompress_native_block(
        $framed, hasher => undef);         # skip checksum verification
    my ($plain, $n) = ClickHouse::Encoder->decompress_native_block(
        $stream, offset => $cursor);       # walk a multi-block stream

Inverse of L</compress_native_block>: verifies the checksum (unless
C<hasher =E<gt> undef>), unpacks the payload by method tag, and
returns the raw Native bytes. In list context also returns the
number of bytes consumed from C<$bytes> (16 + 9 + payload length),
so the caller can advance an offset cursor through a stream of
back-to-back compressed blocks.

=head1 TYPES

=head2 Supported

=over 4

=item *

Integers: C<Int8>, C<Int16>, C<Int32>, C<Int64>, C<UInt8>, C<UInt16>,
C<UInt32>, C<UInt64>.

=item *

Floats: C<Float32>, C<Float64>, C<BFloat16> (CH 24.x; 2-byte truncated
Float32). C<Inf>, C<-Inf>, and C<NaN> are preserved.

=item *

Strings: C<String> (length-prefixed bytes), C<FixedString(N)> (N bytes,
null-padded). Both pass the SV's bytes through unchanged: a UTF-8 string
encodes its UTF-8 bytes, a binary blob encodes its bytes, and truncation
is by byte not codepoint.

=item *

Dates: C<Date>, C<Date32>, C<DateTime>, C<DateTime('tz')> (timezone is part
of the schema, not the value), C<DateTime64(P)> with C<P> in 0..9.

=item *

Decimals: C<Decimal32(S)> (S in 0..9), C<Decimal64(S)> (0..18),
C<Decimal128(S)> (0..38), C<Decimal256(S)> (0..76), and
C<Decimal(P, S)> with C<P> in 1..38 (auto-routed to 32/64/128).

=item *

Enums: C<Enum8('a' = 1, ...)>, C<Enum16(...)>.

=item *

C<Bool> / C<Boolean> (1 byte; truthy/falsy in Perl sense).

=item *

C<UUID> (16 bytes; accept either the standard
C<xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx> string or 16 raw bytes).

=item *

C<IPv4> (UInt32 LE; accept dotted-quad string or integer),
C<IPv6> (16 bytes network-order; accept colon-hex string or 16 raw bytes).

=item *

C<Map(K, V)> (wire-equivalent to C<Array(Tuple(K, V))>; accept either a
hashref or an arrayref of pairs).

=item *

C<Variant(T1, T2, ...)> (CH 24.1+; tagged union). Each row is either
C<undef> (null) or C<[$variant_idx, $value]>, where C<$variant_idx> is
the 0-based position in the declared C<Variant(...)> type list (so
C<Variant(String, UInt32)> uses 0 for String and 1 for UInt32). Up to 254
variants. ClickHouse stores Variant arms alphabetically by type name on
the wire; the encoder remaps your declared index transparently, so you
always use declaration order in your code. C<describe table> returns the
arms already alphabetized, and C<for_table> uses that form, so indices
match either way.

=item *

C<LowCardinality(String)>, C<LowCardinality(FixedString(N))>,
C<LowCardinality(Nullable(String))>, C<LowCardinality(Nullable(FixedString(N)))>.

=item *

Geo: C<Point>, C<Ring>, C<LineString>, C<MultiLineString>, C<Polygon>,
C<MultiPolygon> (aliases over C<Tuple>/C<Array>). L</parse_wkt> turns
a Well-Known-Text string into the nested-arrayref input these
columns expect.

=item *

Composites: C<Array(T)>, C<Tuple(T1, T2, ...)>, C<Nullable(T)>. Arbitrary
nesting; C<Nullable(Nullable(T))> is rejected, matching ClickHouse. Named
tuple elements (C<Tuple(a Int32, b String)>) are accepted; the wire
format itself ignores names, but for a named tuple the encoder will
also accept rows as hashrefs (C<<< { a => 1, b => "x" } >>>) in addition
to arrayrefs (C<[1, "x"]>). Missing keys in a hashref encode as the
type's null placeholder.

=item *

C<SimpleAggregateFunction(func, T)>: parsed as plain C<T>; the
aggregation function name only affects how readers consume the column,
not the wire-level binary format. Full C<AggregateFunction> (with
function-specific binary state) is not supported; use
C<SimpleAggregateFunction> where the inner type matches the on-the-wire
representation.

=item *

C<JSON> / C<Object('json')> (the stable CH 24.8+ JSON type). Each row
is a hashref of leaf values; nested hashrefs are auto-flattened into
dotted-path subcolumns matching CH's internal storage (so
C<<< { user => { name => "alice" } } >>> stores under the C<user.name>
path on the wire and round-trips back as the same nested structure
on decode). Leaf value typing is inferred per-path from Perl SV flags:

=over 4

=item *

Numeric SV with the integer flag set but not the float flag
(C<SvIOK && !SvNOK>) -> C<Int64>.

=item *

Numeric SV with the float flag set (C<SvNOK>): non-integer value ->
C<Float64>; a NV that happens to equal an integer (e.g. C<1.0>)
collapses to C<Int64> on the wire, mirroring CH's own C<JSONEachRow>
inference. Round-trip then decodes back as an integer; to keep an
integer-valued float as C<Float64> use a dedicated C<Float64> column
or a C<<< Variant(Float64, ...) >>>. C<NaN> and C<Inf> stay C<Float64>.

=item *

Blessed scalarref into C<JSON::PP::Boolean>, C<JSON::XS::Boolean>,
C<Types::Serialiser::Boolean>, C<Cpanel::JSON::XS::Boolean>, or
C<boolean> -> C<Bool>. Perl 5.36+ native booleans (C<SvIsBOOL>) also work.

=item *

Anything else stringy -> C<String>.

=item *

C<undef> -> null discriminator for that path in that row.

=back

A path's per-row types may differ; ClickHouse's Dynamic sub-column
representation handles the union. Multiple rows of the same path
sharing different types are encoded as a Variant. Each variant kind
is one of: C<Bool>, C<Float64>, C<Int64>, C<String>, or an
C<Array(...)> of those (homogeneous element type per array). Mixed
or nested arrays are rejected with a clear message.

The C<JSON(name Type, name Type, ...)> form pins specific paths to
concrete inner types ("typed paths"). Those paths skip the
Dynamic+Variant wrapping and emit as regular columns, which is
cheaper on the wire and lets you query them as concrete types
server-side without C<toInt64()> / C<toString()> coercion. Path
names may be dotted (e.g. C<JSON(user.id UInt64)>), the type list
is comma-separated, and typed paths are independent of the dynamic
paths (extra keys still go through Variant). Missing typed keys
encode the inner type's default (0 for numerics, empty for String /
Array / Map, null for Nullable). Inner types whose ClickHouse
serialization includes a wire prefix (C<Variant>,
C<LowCardinality>, C<JSON>, C<Dynamic>) are rejected at encoder
construction.

=item *

C<Dynamic> as a standalone column type: same wire format as one
JSON path's Dynamic sub-column without the Object wrapper. Each row
is a scalar leaf (C<Bool> / C<Float64> / C<Int64> / C<String>), an
C<Array(...)> of those, or C<undef> (null). Hashrefs aren't accepted
here - use a JSON column for object-shaped values.

=back

=head2 Not currently supported

C<LowCardinality(T)> for non-string T, C<AggregateFunction> (per-function
binary state).

Heterogeneous arrays as JSON leaves (e.g. C<<< [1, "two"] >>>); only
homogeneous arrays of C<Bool> / C<Float64> / C<Int64> / C<String>
are supported. Arrays-of-objects (C<<< [{...}, {...}] >>>) and
nested arrays (C<<< [[1,2],[3,4]] >>>) are likewise rejected. Future
work.

C<Nested(...)> at the encoder level. ClickHouse splits a C<Nested> column