view release on metacpan or  search on metacpan

lib/ClickHouse/Encoder.pm  view on Meta::CPAN

Croaks on unknown or malformed types, on C<Nullable(Nullable(...))>, on
empty enum names, on enum values out of range, on C<FixedString(0)>, and
on out-of-range C<Decimal*> / C<DateTime64> parameters.

=head2 encode

    my $bytes = $enc->encode(\@rows);

Returns the raw bytes of one Native-format block. C<\@rows> is an arrayref
of arrayrefs; each inner arrayref must have exactly as many elements as the
encoder's columns. Croaks if a row's shape is wrong, if a value can't be
coerced to its column type, or on string parse errors (see L</Value
coercion>).

=head2 encode_columns

    my $bytes = $enc->encode_columns({
        id   => [1, 2, 3],
        name => ['a', 'b', 'c'],
    });

Like L</encode> but takes a column-oriented hashref
C<<< { name => \@values } >>>. Skips the row-to-column permutation step
that C<encode> performs and is slightly faster when your data already
lives in columns. All arrays must have the same length; missing column
names croak.

=head2 encode_into

    $enc->encode_into(\$buffer, \@rows);

Like L</encode>, but appends the bytes to an existing scalar via reference.
Useful for batching multiple blocks into one HTTP request body without
copying.

=head2 encode_to_handle

    $enc->encode_to_handle($fh, \@rows);

Like L</encode> but writes the bytes directly to a Perl filehandle via
C<PerlIO_write>, skipping a copy. Useful when piping to C<clickhouse-client>
or a network socket.

=head2 for_query

    my $enc = ClickHouse::Encoder->for_query($sql, %opts);

Like L</for_table> but for arbitrary C<select>: runs C<describe
($sql)> to discover the column shape and returns a configured encoder.
Useful when the schema doesn't correspond to a real table (CTEs,
joins, computed columns). Options are the same as L</for_table>; the
caller is responsible for the SQL.

=head2 insert_http

    my $resp = ClickHouse::Encoder->insert_http(
        host => 'localhost', port => 8123, table => 'events',
        columns => \@cols, rows => \@rows,
        compress => 'zstd',   # optional
        scheme   => 'https',  # optional; needs IO::Socket::SSL + Net::SSLeay
        settings => {         # optional per-query CH settings
            max_execution_time => 30,
            max_memory_usage   => '10G',
        },
        dedup_token => $batch_id,  # optional idempotency token
    );
    die "insert failed (status $resp->{status}): $resp->{content}"
        unless $resp->{success};
    # CH response metadata, when present, is also attached:
    my $qid     = $resp->{ch}{'query-id'};
    my $written = $resp->{ch}{summary}{written_rows};

Thin convenience wrapper that builds an encoder, encodes rows, and
POSTs the bytes to C<< http(s)://host:port/?query=insert into $table format native >>.
Pass C<encoder => $enc> instead of C<columns> to reuse one. C<compress>
accepts C<'raw'> (default), C<'zstd'>, or C<'gzip'>. C<scheme =E<gt> 'https'>
enables TLS via L<HTTP::Tiny>'s SSL support (install
L<IO::Socket::SSL> and L<Net::SSLeay>). C<ssl_options> / C<verify_SSL>
pass through to L<HTTP::Tiny>. C<settings> applies per-query CH
settings via URL params. C<dedup_token> stamps the request with an
C<insert_deduplication_token> so an identical retry is rejected
server-side. Returns the L<HTTP::Tiny> response hashref with a
C<ch =E<gt> { query-id, server, format, exception-code, summary,
progress, ... }> slot containing parsed
C<X-ClickHouse-*> response headers; no automatic retries (do HTTP
policy in the caller, or use L</bulk_inserter>).

=head2 for_table

    my $enc = ClickHouse::Encoder->for_table($table, %opts);

Convenience constructor that introspects a table's schema and runs

    describe table $table format tabseparated

C<$table> must be a plain identifier or C<db.table>; each component
matches C<[A-Za-z_][A-Za-z0-9_]*>. Anything else is rejected to avoid
SQL injection through the C<describe table> query.

Options (all optional):

=over 4

=item C<via>

C<'client'> (default) shells out to C<clickhouse-client>. C<'http'> uses
L<HTTP::Tiny> against C<host:port> directly with no external binary
dependency. Recommended on environments without C<clickhouse-client>.

=item C<host>, C<port>, C<database>, C<user>

Connection parameters. Defaults: C<localhost>; C<9000> for
C<<< via => 'client' >>> or C<8123> for C<<< via => 'http' >>>;
C<default>, C<default>.

=item C<password>

For C<client> mode, passed via C<CLICKHOUSE_PASSWORD> env var. For C<http>
mode, sent as the C<X-ClickHouse-Key> header.

=item C<client>

Path to the C<clickhouse-client> executable (only for
C<<< via => 'client' >>>). Default: whatever C<exec> finds on C<$PATH>.

=item C<scheme>, C<timeout>, C<ssl_options>, C<verify_SSL>, C<settings>

For C<<< via => 'http' >>>: the URL scheme (default C<http>; pass
C<https> for TLS), request timeout in seconds (default 10),
optional SSL options / verify_SSL passthrough to L<HTTP::Tiny>,
and a hashref of per-query CH settings. Ignored under
C<<< via => 'client' >>> (the C<clickhouse-client> binary handles
its own connection).

=back

=head2 stream

lib/ClickHouse/Encoder.pm  view on Meta::CPAN

C<max_execution_time> and similar). C<dedup_token> is meaningful
only on insert and is ignored if passed here.

=head2 bulk_inserter

    my $bi = ClickHouse::Encoder->bulk_inserter(
        host => 'db.example', port => 8123, table => 'events',
        columns => \@cols, batch_size => 5000, compress => 'zstd',
        retries => 3);
    $bi->push([$row]) for @rows;
    $bi->finish;

Holds an L<HTTP::Tiny> instance with keep-alive across batches,
accumulates rows, auto-flushes at C<batch_size>, retries transient
HTTP failures (5xx and 599 network errors) with exponential backoff
and jitter. 4xx errors die immediately. Options:

=over 4

=item C<host>, C<port>, C<database>, C<user>, C<password>, C<scheme>, C<timeout>

Same as L</for_table>; passed to L<HTTP::Tiny>.

=item C<table>

Required; same identifier rule as L</for_table>.

=item C<encoder> or C<columns>

Pass either an existing encoder or a column list (used to build one).

=item C<batch_size>

Auto-flush threshold (default 10_000).

=item C<retries>

Max retries on transient failure (default 3). Set to 0 to disable.

=item C<retry_wait>

Base backoff in seconds (default 0.5). Waits grow exponentially -
the window for attempt I<n> is C<retry_wait * 2 ** n> - with equal
jitter (a random point in the upper half of the window), so
concurrent inserters retrying the same failed server do not
resynchronise into a thundering herd.

=item C<retry_max_wait>

Upper bound in seconds on the backoff window (default 30), capping
the exponential growth from C<retry_wait>.

=item C<compress>

C<'raw'> (default), C<'zstd'>, or C<'gzip'>. Sets the
C<Content-Encoding> header accordingly.

=item C<scheme>, C<ssl_options>, C<verify_SSL>

C<scheme =E<gt> 'https'> enables TLS via L<HTTP::Tiny> (install
L<IO::Socket::SSL> and L<Net::SSLeay>). C<ssl_options> and
C<verify_SSL> pass through to L<HTTP::Tiny>.

=item C<settings>

Hashref of per-query CH settings (C<max_execution_time>,
C<max_memory_usage>, ...) appended to every flush as URL params.

=item C<dedup_token>

Stamps every POST with C<insert_deduplication_token>. Identical
retries are rejected server-side, making the inserter
transactionally idempotent.

=back

Methods on the returned object:

=over 4

=item C<<< $bi->push($row) >>>

Append a single row. Auto-flushes if buffer crosses C<batch_size>.

=item C<<< $bi->push_many(\@rows) >>>

Append many rows in one call.

=item C<<< $bi->flush >>>

Flush whatever is in the buffer (idempotent: no-op when empty).

=item C<<< $bi->finish >>>

Flush and return C<<< { rows => $sent_total, batches => $batches } >>>.

=item C<<< $bi->buffered_count >>> / C<<< $bi->sent_rows >>> / C<<< $bi->sent_batches >>>

Instrumentation accessors.

=item C<<< $bi->summary >>>

Hashref of cumulative C<X-ClickHouse-Summary> stats rolled up across
batches (C<written_rows>, C<written_bytes>, C<elapsed_ns>, ...).

=item C<<< $bi->last_response >>>

The L<HTTP::Tiny> response hashref from the most recent flush, with
the parsed C<ch =E<gt> { query-id, server, format, exception-code,
summary, progress, ... }>
slot attached (whichever C<X-ClickHouse-*> headers the server sent).
C<undef> until the first flush succeeds.

=back

=head2 decimal128_str

    my $s = ClickHouse::Encoder->decimal128_str($lo, $hi, $scale);

Convert a decoded C<Decimal128> low/high uint64 pair (as returned
by L</decode_block>) into a signed decimal string with the given