Fetch

 view release on metacpan or  search on metacpan

lib/Fetch.pm  view on Meta::CPAN

use File::Raw::JSON ();   # JSON encode/decode via its C ABI (ft_json.h / _abi_ptr)

require XSLoader;
XSLoader::load('Fetch', $VERSION);

require Fetch::Future;
require Fetch::Loop;
require Fetch::Loop::Standalone;
require Fetch::Response;
require Fetch::Headers;
require Fetch::CookieJar;
require Fetch::WebSocket;

1;

__END__

=head1 NAME

Fetch - HTTP/2 Future-based user agent

=head1 VERSION

Version 0.16

=head1 SYNOPSIS

    use Fetch;

    # works out of the box - no event loop to set up
    my $res = Fetch->new->get('https://example.com')->get;
    print $res->content if $res->is_success;

    # every call returns a Fetch::Future; ->get awaits it
    my $ua  = Fetch->new(timeout => 10);
    my $f   = $ua->post('https://api/things',
                        headers => { 'Content-Type' => 'application/json' },
                        body    => '{"name":"x"}');
    my $res = $f->get;

    # many requests concurrently on one loop
    my @futs = map { $ua->get($_) } @urls;
    Fetch::Future->needs_all(@futs)->get;
    my @bodies = map { $_->get->content } @futs;

    # a live WebSocket
    my $ws = $ua->websocket('wss://host/socket')->get;
    $ws->send('hello');
    my $reply = $ws->next_message->get;

=head1 DESCRIPTION

Fetch is an HTTP user agent whose socket, TLS, HTTP/2 framing and HTTP/1.1
parsing hot path lives in vendored C, and whose asynchronous results are
L<Fetch::Future> objects that compose with the Hyperman event loop and other
CPAN loops (IO::Async, AnyEvent) - or with nothing at all, since Fetch ships
its own event loop (L<Fetch::Loop::Standalone>) and uses it automatically.

HTTP/1.1 and HTTP/2 (ALPN-negotiated over TLS), over cleartext and TLS, with
keep-alive connection pooling, redirect following, per-request timeouts,
streaming response bodies, a cookie jar, JSON request/response helpers, and
native WebSockets. Every request method returns a
L<Fetch::Future>; C<< ->get >> on one awaits it, pumping whichever event loop
is active (its own if none), so the same code serves both a simple synchronous
call and thousands of requests multiplexed on one loop.

=head1 CONSTRUCTOR

=head2 new(%args)

    my $ua = Fetch->new(
        timeout    => 10,
        tls_verify => 1,
        headers    => { 'Accept' => 'application/json' },
        cookie_jar => 1,
    );

Create a user agent. All arguments are optional:

=over 4

=item C<loop>

The event loop to run on. Omit it and Fetch uses its own
L<Fetch::Loop::Standalone> (so C<< ->get >> just works with no framework). Pass
a raw L<IO::Async::Loop> or L<Hyperman::Loop> and it is wrapped automatically;
pass the string C<'AnyEvent'> to drive AnyEvent; or pass a ready-made
L<Fetch::Loop> adapter. See L</"EVENT LOOPS">.

Omitting it gives every such agent the B<same> loop - one implicit
L<Fetch::Loop::Standalone> per process, rebuilt automatically in a child after
a fork. Agents therefore multiplex: awaiting a request on one drives whatever
the others have in flight rather than stalling them. Pass an explicit loop to
opt out and get an isolated one.

=item C<headers>

Default headers sent on every request, as a hashref, an arrayref of
C<< name => value >> pairs (duplicates preserved), or a L<Fetch::Headers>.
Per-request C<headers> are merged on top (see L</"REQUEST OPTIONS">).

=item C<agent>

The C<User-Agent> string. Defaults to C<"Fetch/$VERSION">.

=item C<tls_verify>

Whether to verify the peer certificate and hostname for C<https>. Default true.
Overridable per request.

=item C<timeout>

Default per-request deadline in seconds (fractional allowed). C<0> (the
default) means no timeout. Overridable per request.

=item C<max_redirects>

How many redirects to follow. Default C<5>; C<0> disables following.
Overridable per request.

=item C<keep_alive>

Reuse connections via a keep-alive pool (default true). Set false to close
every connection after one request.

=item C<pool_size>

Maximum idle connections the keep-alive pool parks (default C<32>).

=item C<cookie_jar>

A L<Fetch::CookieJar> to store and send cookies (applied across redirects), or
a true scalar to create a fresh one. Default: no jar.

=item C<simple_response>

Resolve requests to a plain unblessed hashref C<< { status => ..., headers =>
[k, v, ...], content => ... } >> instead of a blessed L<Fetch::Response>. Read
fields directly (C<< $res->{status} >>, C<< $res->{content} >>) rather than via
methods. This skips the response object's method dispatch on the hot path; the
saving is small (a couple of percent) and only shows when you actually read the
response, so reach for it when you are consuming millions of responses and want
the leanest possible per-response cost. Default false.

=back

=head1 REQUEST METHODS

Each returns a L<Fetch::Future> that resolves to a L<Fetch::Response> (or fails
with an error string). They never block; call C<< ->get >> on the future to
await the result.

=head2 get / head / delete

    my $f = $ua->get($url, %opt);

=head2 post / put

    my $f = $ua->post($url, body => $bytes, %opt);

=head2 request($method, $url, %opt)

    my $f = $ua->request('PATCH', $url, body => $bytes, %opt);

The general form the verb helpers dispatch to; use it for any method.

=head1 REQUEST OPTIONS

Passed as a trailing C<< key => value >> list to any request method:

=over 4

=item C<headers>

Extra headers for this request - a hashref, an arrayref of pairs (keeping
duplicate names, e.g. multiple C<X-*> values), or a L<Fetch::Headers>. Each
named field overrides the agent default of the same name.

=item C<body>

The request body (bytes). Content-Length is added automatically unless you
set it yourself.

=item C<json>

A Perl data structure to send as a JSON body: it is encoded and
C<Content-Type: application/json> is set (unless you gave your own). Takes
precedence over C<body>. Pair it with L<Fetch::Response/json> to decode the

lib/Fetch.pm  view on Meta::CPAN

=item C<timeout>

Override the agent timeout for this request (seconds; C<0> disables it).

=item C<tls_verify>

Override certificate/hostname verification for this C<https> request.

=item C<max_redirects>

Override how many redirects this request follows.

=item C<on_body>

A coderef called with each body chunk as it arrives, instead of buffering.
Suits large downloads and server-sent events: the buffer is compacted so an
endless stream does not grow memory, and the resolved response body is empty.

    $ua->get($url, on_body => sub { my ($chunk) = @_; print $chunk })->get;

=item C<on_headers>

A coderef called once, as soon as the response status line and headers have
been parsed - before any C<on_body> chunk - with the status code and the header
list:

    $ua->get($url,
        on_headers => sub { my ($status, $headers) = @_; ... },  # $headers: [k,v,...]
        on_body    => sub { my ($chunk) = @_; ... },
    )->get;

This lets a streaming consumer act on the status and headers up front (for
example, to open a downstream writer) rather than waiting for the whole
response. Not fired for a WebSocket upgrade.

=back

=head1 WEBSOCKETS

=head2 websocket($url, %opt)

    my $ws = $ua->websocket('ws://host/echo')->get;   # or wss://

Open a WebSocket (RFC 6455). Returns a L<Fetch::Future> that resolves, after
the C<101> handshake, to a L<Fetch::WebSocket> for sending and receiving
messages. Accepts C<ws://>/C<wss://> (and C<http>/C<https>); C<tls_verify> and
C<timeout> options apply to the handshake.

=head2 clone(%overrides)

    my $shared  = Fetch->new;                        # built once
    my $scoped  = $shared->clone(cookie_jar => 1);   # its own jar, same pool

Another agent over this one's connection pool and event loop, with the given
options replaced. Everything not named is inherited.

The case it exists for is a B<per-request cookie jar>. A jar belongs to the
agent, so a jar on a long-lived agent is shared by every request that agent
serves: fine when the cookies authenticate the application itself, a
cross-request leak the moment they identify an end user. Building a whole fresh
agent per request avoids the leak but throws away the keep-alive pool and
re-resolves the loop adapter, which is most of what C<new> costs. A clone gives
you the isolation without the bill.

C<cookie_jar>, C<headers>, C<agent>, C<timeout>, C<tls_verify>,
C<max_redirects>, C<keep_alive> and C<simple_response> can be overridden;
C<< cookie_jar => undef >> drops an inherited jar. Headers are copied rather
than shared, so a clone cannot write into the parent's defaults.

C<loop> and C<pool_size> cannot be overridden and croak if given: they are what
the clone shares, and an agent on a different loop driving the parent's parked
connections would be reaching into the wrong one.

A clone holds references to the shared pool and loop, so they live until the
last agent using them goes. Keep the parent alive for as long as its clones, as
you would anyway.

=head1 ACCESSORS

=head2 loop

The event-loop adapter this agent runs on.

=head2 cookie_jar

The L<Fetch::CookieJar> in use, or undef.

=head1 OBSERVING OUTBOUND REQUESTS

=head2 Fetch->on_request(\&start, \&done)

Watch every request this B<process> makes, including ones it did not write.

    Fetch->on_request(
        sub {
            my ($method, $url, $headers) = @_;
            push @$headers, traceparent => current_span_id();
            return { at => time, url => $url };        # the token
        },
        sub {
            my ($token, $res, $err) = @_;
            record($token->{url}, time - $token->{at},
                   $err ? "failed: $err" : $res->status);
        },
    );

C<start> fires just before a request goes out, with the merged header list
still B<mutable>: push a name and a value onto C<$headers> and the request
carries them. That is the point of the hook - L</REQUEST OPTIONS>' own
C<headers> is set by whoever made the call, which is no use to a tracing
layer that has to annotate a request the application wrote.

Whatever C<start> returns is the B<token>, handed back to C<done> when that
same request settles. It is an ordinary Perl scalar - a hashref of what you
want to remember is the usual thing - so the two halves correlate without a
lookup table keyed on something that might repeat.

C<done> fires exactly once for every C<start>, with exactly one of C<$res>
(a L<Fetch::Response>) and C<$err> (the failure message). That includes a
timeout, a refused connection and a DNS failure, which are the endings an
instrumented client most wants and the easiest to leak. C<done> is optional.

Three things to know, all of which are the contract rather than an oversight:

=over 4

=item *

B<Per hop, not per call.> A redirect chain is several requests, each of which
went somewhere, and each is observed. One C<< ->get >> that follows two
redirects fires C<start> three times.

=item *

B<Registration is process-global and permanent.> Not per agent: an agent is a

lib/Fetch.pm  view on Meta::CPAN

    use IO::Async::Loop;
    my $loop = IO::Async::Loop->new;
    my $ua   = Fetch->new(loop => $loop);       # shares this loop
    my $f    = $ua->get('https://example.com/');
    $loop->loop_once until $f->is_ready;
    my $res  = $f->get;

Supported loops: the built-in L<Fetch::Loop::Standalone>, plus
L<Fetch::Loop::IOAsync>, L<Fetch::Loop::AnyEvent> and L<Fetch::Loop::Hyperman>.

A request future remembers the loop it was issued on, and every future derived
from it by C<then>/C<followed_by>/C<transform> inherits that. So C<< ->get >>
always pumps the loop that owns the socket, however many agents and loops the
program has, and in whatever order they were built. Awaiting a future on a loop
that was never going to resolve it - one with no watchers and no timers - dies
with a diagnostic rather than blocking in the kernel forever.

=head1 C ABI

Fetch exposes a small C ABI so another XS module can drive it from C, with no
per-request Perl round trip on the hot path. It is how L<Reverse::Proxy> builds
its upstream requests entirely in C. This is not part of the Perl API - if you
are writing Perl, use the request methods above; the ABI is only for XS
consumers.

The ABI is a versioned function pointer table. A consumer vendors a copy of the header C<fetch_abi.h> (shipped
in this distribution under C<include/fetch/>) at a pinned C<FETCH_ABI_VERSION>, then at boot resolves the table and checks the version:

    #include "fetch_abi.h"   /* vendored, after the perl.h includes */

    /* in BOOT: */
    IV p = 0;
    if (call_pv("Fetch::_abi_ptr", G_SCALAR) > 0) { SPAGAIN; p = POPi; PUTBACK; }
    const fetch_abi *FETCH = NULL;
    if (p) {
        const fetch_abi *a = INT2PTR(const fetch_abi *, p);
        if (a && a->abi_version >= FETCH_ABI_VERSION) FETCH = a;
    }
    /* FETCH == NULL means the running Fetch is too old; the consumer decides
     * whether that is a hard error or a Perl fallback. The check is >=, never
     * ==: the table only grows at the end, so a newer Fetch still holds every
     * entry the consumer was written against. */

=head2 Fetch::_abi_ptr

Returns the address of Fetch's C<fetch_abi> table as an C<IV>. Call it once, at
boot, and C<INT2PTR> the result to a C<const fetch_abi *>. A version mismatch
must never be treated as a crash - compare C<abi_version> first and fall back.

=head2 The table

C<fetch_abi> (see C<fetch_abi.h> for the exact signatures and ownership rules)
holds, after C<abi_version>:

=over 4

=item C<ua_new(kv, nkv)>

Construct a Fetch user agent from C<nkv> flat key/value SVs (the same options
C<new> takes: C<loop>, C<pool_size>, C<tls_verify>, C<timeout>, C<agent>,
C<headers>, C<cookie_jar>, C<keep_alive>, C<max_redirects>,
C<simple_response>). Returns the blessed Fetch UA SV (+1 owned). A consumer may
instead call C<< Fetch->new >> from Perl once and cache the object; C<ua_new>
just removes that last Perl call.

=item C<request(ua_sv, method, url, hdrs, nhdrs, body, blen, timeout, max_redirects, map, ud)>

Issue one HTTP request on C<ua_sv>, building it entirely from C. The headers
are a flat C<fetch_hdr> array; C<max_redirects> below zero means "use the UA
default". Returns a L<Fetch::Future> SV (+1 owned) - hand it to an awaiting
server or call C<< ->get >> on it. When the request settles, your C<map>
callback shapes the value the future resolves to (for a proxy, a PSGI
C<[ status, \@headers, \@body ]>).

=item C<res_parts(res, status, headers, body)>

Pull the status, the flat header AV and the content SV out of an
already-resolved response (a L<Fetch::Response> or a C<simple_response> hash)
with no method dispatch. Any out-pointer may be C<NULL>; the returned C<headers>
and C<body> are borrowed. For the blocking (awaited) path.

=item C<request_stream(ua_sv, method, url, hdrs, nhdrs, body, blen, timeout, max_redirects, on_headers, on_body, on_done, ud)>

Like C<request>, but streams the response instead of buffering it: C<on_headers>
fires once up front with the status and header AV, C<on_body> once per body
chunk, and C<on_done> at completion (with success/failure). Returns the request
future SV (+1 owned) - keep it alive until C<on_done> fires. Lets a consumer
forward a large download or an endless SSE stream with flat memory.

=item C<tunnel_connect(host, port, tls, verify)> and friends

A raw blocking upstream TCP connection Fetch owns, for a proxy's
Upgrade/WebSocket tunnel where the consumer splices bytes both ways itself.
When C<tls> is true the connection reuses Fetch's own client C<SSL_CTX> (SNI,
and hostname/certificate verification when C<verify> is true), so the consumer
tunnels to a C<wss>/C<https> upstream without linking OpenSSL. This is what lets
L<Reverse::Proxy> tunnel to a TLS upstream.

Unlike the rest of the table, these six entries are pure C - they take no
C<pTHX> and touch no SV, so they can be called directly from inside a
C<select()> splice loop:

=over 4

=item *

C<tunnel_connect(host, port, tls, verify)> - open the connection; returns an
opaque handle, or C<NULL> on failure (DNS, connect, or TLS handshake).

=item *

C<tunnel_fd(conn)> - the underlying socket fd, to hand to C<select()>/C<poll()>.

=item *

C<tunnel_read(conn, buf, len)> - read bytes; returns the count (E<gt>0), C<0> at
EOF, or C<-1> on error.

=item *

C<tunnel_write_all(conn, buf, len)> - write the whole buffer; returns C<0> when
all bytes are written, C<-1> on error.

=item *

C<tunnel_pending(conn)> - bytes already buffered inside the TLS layer; drain
these before trusting C<select()> readiness (always C<0> for a plain
connection).

=item *

C<tunnel_close(conn)> - shut the connection down and free the handle.

=back

=item C<on_request(start, done, ud)>  I<(v2)>

Observe outbound requests.

    static void *start(pTHX_ const char *method, STRLEN mlen,
                       const char *url, STRLEN ulen, AV *headers, void *ud);
    static void  done(pTHX_ void *token, SV *res, SV *err, void *ud);

C<start> fires once per B<hop> with the merged header list still B<mutable> -
push a name and a value onto C<headers> and the request carries them. That is
what the hook is for: L</REQUEST OPTIONS>' C<headers> is set by the caller,
which is no use to anything that wants to annotate a request the application