EV-ClickHouse
view release on metacpan or search on metacpan
to a backup host on specific errors).
- on\_progress => sub { my ($rows, $bytes, $total\_rows, $written\_rows, $written\_bytes) = @\_ }
Called on native protocol progress packets. Not fired for HTTP.
- on\_disconnect => sub { }
Called when an established connection closes (by `finish`, server
disconnect, or mid-flight error). Only fires if `on_connect` had
previously fired - it does **not** fire for connect-phase failures
(refused, timeout, ServerHello stall) since no connection was ever
established. Fires after internal state has been reset, so it is safe
to queue new queries or call `reset` from inside the handler.
- on\_trace => sub { my ($message) = @\_ }
Debug trace callback. Called with internal state-machine messages
(connect, dispatch, disconnect). Useful for diagnosing protocol issues.
- on\_failover => sub { my ($old\_host, $old\_port, $new\_host, $new\_port, $msg) = @\_ }
Multi-host only. Fires after the failover wrapper rotates to the next
host in the `hosts => [...]` list, with the old and new (host, port)
pair plus the triggering error message. Use it for metrics ("which host
am I on?") or to log host transitions. Fires before the user's `on_error`.
**Options:**
- compress => 0 | 1
Enable compression: gzip on HTTP (request and response), LZ4 with CityHash
checksums on the native protocol. Default: `0`. Native compression
requires liblz4 at build time.
- session\_id => $id
HTTP session id for stateful operations (temporary tables, SET, etc.).
Native protocol has stateful sessions intrinsically; this option is HTTP-only.
- connect\_timeout => $seconds
TCP/TLS connection timeout. `0` (default) means no timeout. Floating
point allowed.
- query\_timeout => $seconds
Default per-query timeout applied to every query and insert. The query
callback receives a `timeout` error if exceeded. Override per-call via
the `query_timeout` key in the settings hashref.
- max\_query\_size => $bytes
Client-side guard: croak before sending any query whose SQL text exceeds
this many bytes. `0` (default) disables the check. Useful as a
last-resort defense against accidentally sending unbounded strings.
- max\_recv\_buffer => $bytes
Defensive ceiling on the response. The cap applies to the raw recv
buffer (every protocol), the chunked-decoded body (HTTP), and the
gzip-decompressed body (HTTP), so the same upper bound applies to the
user-visible payload regardless of transport encoding. On overflow the
query callback receives an appropriate error ("recv buffer overflow",
"chunked response too large", or "gzip body exceeds max\_recv\_buffer")
and the connection is torn down so no subsequent query can slip past
the cap on the same socket. `0` (default) keeps the historical
no-cap behaviour (still bounded internally by a hard 128 MB ceiling
on compressed paths). Recommended in production when the schema is
constrained and you want a hard upper bound (e.g.
`128 * 1024 * 1024` for 128 MB).
- http\_basic\_auth => 0 | 1
HTTP only. When set, send credentials as
`Authorization: Basic base64(user:password)` instead of the default
`X-ClickHouse-User` / `X-ClickHouse-Key` header pair. Use this when
the connection passes through an HTTP gateway (nginx, Envoy, ...) that
strips the X-ClickHouse-\* headers but forwards Basic auth verbatim.
Default: `0`.
- auto\_reconnect => 0 | 1
Reconnect automatically on connection loss. Default: `0`. When enabled,
queued (unsent) queries are preserved across reconnects; in-flight queries
receive an error.
The reconnect path covers TCP/TLS connect failures, `connect_timeout`
or `query_timeout` expiry, and any clean server-side EOF (idle or
mid-request). Mid-query I/O errors (ECONNRESET / EPIPE) and a malformed
native ServerHello are **not** retried - they typically indicate a
misconfigured peer or client-side bug that retry would only loop on.
Combine with `reconnect_max_attempts` for an explicit ceiling.
- settings => \\%hash
ClickHouse settings applied to every query and insert. Per-call settings
(see ["query"](#query), ["insert"](#insert)) override these.
settings => { async_insert => 1, max_threads => 4 }
- keepalive => $seconds
Send a keepalive request every N seconds while the connection is idle:
a native CLIENT\_PING on the native protocol or a `GET /ping` on HTTP
(some load balancers / NATs drop idle HTTP connections after a few
seconds; TCP-level keepalive is too coarse). Default: `0` (disabled).
- reconnect\_delay => $seconds
Initial delay for the `auto_reconnect` exponential backoff. Each failed
attempt doubles the delay, capped at `reconnect_max_delay`. Default:
`0` (immediate retry, no backoff).
- reconnect\_max\_delay => $seconds
Backoff ceiling. Default: `0`, meaning no explicit cap; the implementation
still bounds the backoff exponent at 20 doublings, so with
`reconnect_delay = 0.5` the worst case is roughly 6 days. Setting an
explicit ceiling is recommended in production.
- `raw => 1`
HTTP only. The callback receives the raw response body as a scalar string
instead of parsed rows. Use with an explicit `format` clause:
$ch->query("select * from t format CSV", { raw => 1 }, sub {
my ($body, $err) = @_;
});
Croaks if used with the native protocol.
- `query_timeout => $seconds`
Per-query timeout, overriding the connection-level `query_timeout`.
- `on_data => sub { my ($rows) = @_; ... }`
Native protocol only. A code ref called for each data block as it arrives,
for streaming large result sets. Rows are delivered incrementally and
**not** accumulated, so the final callback receives `(undef)` rather than
all rows. The final callback always fires on completion or error, even if
no data block was emitted (empty result, server-side error before the
first block).
$ch->query("select * from big_table",
{ on_data => sub { my ($rows) = @_; process_batch($rows) } },
sub { my (undef, $err) = @_; warn $err if $err },
);
- `external => \%tables`
Native protocol only. Ships one or more in-memory data blocks that the
query can reference as tables, JOIN against, or filter with `IN` -
without creating a server-side temporary table. Each entry maps a table
name to `{ structure => [...], data => [...] }`:
$ch->query(
"select u.id, u.name from users u where u.id in _wanted",
{ external => {
_wanted => {
structure => [ id => 'UInt64' ],
data => [ [7], [42], [911] ],
},
} },
sub { my ($rows, $err) = @_; ... },
);
`structure` is a flat list of `name => type` pairs (ClickHouse type
names, e.g. `UInt64`, `String`, `Float64`); `data` is an arrayref of
row arrayrefs, encoded with the same type machinery as ["insert"](#insert). An
empty `data` arrayref is a valid zero-row table. Several external tables
may be supplied at once. Croaks on the HTTP protocol or on a malformed
spec (odd structure list, non-arrayref row, or a column type that
cannot be encoded).
**Native protocol type notes:** values come back as typed Perl scalars.
By default `Date`/`DateTime` are integers (days since epoch / Unix
timestamps); enable `decode_datetime` for strings. `Enum` values are
numeric codes; `decode_enum` returns labels. `Decimal` values are
unscaled integers; `decode_decimal` scales them to floats.
`SimpleAggregateFunction` is transparently decoded as its inner type.
`Nested` columns become arrays of tuples. `LowCardinality` works
correctly across multi-block results with shared dictionaries.
## insert
$ch->insert($table, $data, sub { my (undef, $err) = @_ });
$ch->insert($table, $data, \%settings, sub { my (undef, $err) = @_ });
`$data` may be either:
- A pre-formatted TabSeparated string (tabs separate columns,
newlines separate rows, with the standard ClickHouse escapes).
- An arrayref of arrayrefs (rows of column values).
When using arrayrefs, no TSV escaping is needed: `undef` maps to null
and strings may contain tabs and newlines freely.
Nested arrayrefs (Array/Tuple columns) and hashrefs (Map columns) are
supported **only on the native protocol**, where the encoder has the
column type from the server's sample block. On HTTP the same call
croaks rather than silently produce malformed TSV; use the native
protocol or pre-serialise nested types into ClickHouse TSV literal form.
# Native: nested types encode directly.
$ch->insert("my_table", [
[1, "hello\tworld"], # embedded tab
[2, undef], # null
[3, [10, 20]], # Array column (native only)
[4, { a => 1, b => 2 }], # Map column (native only)
], sub { ... });
The optional `\%settings` hashref works exactly as in ["query"](#query),
including `query_id`, `query_timeout`, and `params`. Two extra
flags are recognised here:
- `idempotent => 1 | $token`
Auto-mints (or uses the supplied) `insert_deduplication_token`, so a
reconnect-driven retry of the same insert doesn't double-write. Falsy
values are a no-op.
- `async_insert => 1`
Enables ClickHouse server-side insert batching by setting
`async_insert=1, wait_for_async_insert=0`. Both sub-settings can be
overridden by passing them explicitly.
## ping
$ch->ping(sub { my ($result, $err) = @_ });
Send a no-op round trip to verify the connection is alive. On success
`$result` is true, `$err` is `undef`. On error: `(undef, $error)`.
## is\_healthy
$ch->is_healthy(sub { my ($ok, $err) = @_ });
$ch->is_healthy(sub { ... }, $timeout_seconds);
Bounded health probe: wraps ["ping"](#ping) with a deadline (default 5s). The
pending\_count, ...) and `$ch->pending_queries` lists the
in-flight + queued entries with their query\_ids and age. Both are
read-only debug accessors - safe to call from a signal-handler-style
dump path.
- 8. Don't fight the freelist
The XS layer keeps freelists for both cb\_queue and send\_queue entries,
so allocating callbacks is essentially free after warm-up. The
implication: avoid wrapping the connection in heavy wrappers that
clone the connection per call - there is no per-call setup cost worth
amortising away.
# ARCHITECTURE
The client is a single state machine driven by an [EV](https://metacpan.org/pod/EV) event loop. Each
connection holds: a TCP fd (non-blocking), a send buffer, a receive
buffer, a callback queue (next-in-line per protocol), and a pending
send queue (buffered before connect).
State transitions:
Connect TCP --> [TLS handshake] --> [Native ServerHello]
--> Connected --> { dispatch from send_queue;
parse response; deliver via cb_queue }
The connect\_timeout timer covers all three pre-Connected stages.
auto\_reconnect re-runs the chain via `schedule_reconnect`.
Two key invariants:
- Native protocol is strictly request/response. Only one query is
in-flight per connection at a time. `insert_streamer` serialises
batches against this constraint.
- `callback_depth` guards against `self` being freed mid-callback.
Every callback dispatch increments it; `check_destroyed` defers the
final `Safefree` until depth returns to zero.
For deeper detail (state-machine table, queue semantics) see `CLAUDE.md`
in the source distribution.
# TYPES
Per-column wire format and Perl-side gotchas. All numeric types
round-trip stable raw values by default; opt into string forms via
`decode_datetime`, `decode_decimal`, `decode_enum`.
- Integers
Int8..Int64 / UInt8..UInt64: native Perl IV/UV. Int128/UInt128/Int256/UInt256
return decimal string representations on platforms with `__int128` (Int128/UInt128)
or always for the 256-bit forms.
- Floats
Float32/Float64 round-trip exactly within IEEE-754 limits. `NaN`/`+Inf`/
`-Inf` are preserved.
- BFloat16
Top 16 bits of a Float32. Encoded by truncation; decoded by zero-extension.
Suitable for ML feature columns; not for accounting.
- Decimal32/64/128
Decoded as IV (raw integer) or NV (scaled to N decimal digits if
`decode_decimal => 1`). Decimal128 over very long precision may lose
trailing digits in the NV form; pass `decode_decimal => 0` and divide
yourself with [Math::BigInt](https://metacpan.org/pod/Math%3A%3ABigInt) for exact arithmetic.
- Decimal256
Returns raw 32 LE bytes. Decode with [Math::BigInt](https://metacpan.org/pod/Math%3A%3ABigInt) (see
`eg/decimal_bigmath.pl`).
- Date / Date32 / DateTime / DateTime64
Default: integer (days since epoch / Unix seconds). With `decode_datetime`:
`YYYY-MM-DD` or `YYYY-MM-DD HH:MM:SS` or `YYYY-MM-DD HH:MM:SS.ffffff`.
DateTime carries a timezone string; the formatted output uses it.
- Bool
Decoded as 0/1. Encoded from any truthy/falsy SV. ClickHouse stores
internally as UInt8 0/1.
- String / FixedString
Bytes-in, bytes-out. No UTF-8 transformation.
- UUID
Canonical hex form `xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx`. Encode
accepts the same.
- IPv4 / IPv6
Dotted-quad / canonical IPv6 strings.
- Enum8 / Enum16
Default: integer code. With `decode_enum => 1`: label string.
- Nullable(T)
`undef` in Perl maps to null; otherwise the inner type's encoding.
- Array(T)
Perl arrayref of inner-type values.
- Tuple(T1, T2, ...)
Perl arrayref ordered as the type declaration. Named tuples
(`Tuple(a Int32, b String)`) are still arrayref-positional;
parse the name from `column_types` if you need it.
- Map(K, V)
Perl hashref. Keys are stringified.
( run in 0.444 second using v1.01-cache-2.11-cpan-f4a522933cf )