view release on metacpan or  search on metacpan

README.md  view on Meta::CPAN

'postgresql')`. A `db.statement` you put in a database span's `data` yourself is masked the same way.

### Following a request across services

Traces use the [W3C Trace Context](https://www.w3.org/TR/trace-context/) standard (a `traceparent`
header), so an error or a slow call can be followed from one service into the next.

**Incoming**: automatic with `PSGIPerformance` or `Dancer2Performance`. A request that arrives with
a valid `traceparent` header continues that trace (same trace id), and its root span records the
caller's span as its parent. A missing or malformed header just starts a new trace.

**Outgoing**: wrap each HTTP call you make during a request in `http_span`. It records the call as
an `http` span named after the method and host (never the path or query) and hands your code a
hashref of headers to add; the header's parent id is that span's own id, so the called service's
spans nest under it. It works with any HTTP client, records the span even when the call dies
(re-raising unchanged), and returns what your code returned:

```perl
use HTTP::Tiny;

my $http = HTTP::Tiny->new;
my $response = ForgeOps::Tracker::http_span(POST => $url, sub {
    my ($headers) = @_; # { traceparent => '00-...' }
    $http->post($url, { headers => { %$headers, 'Content-Type' => 'application/json' }, content => $body });
});
```

Outside a request your code gets an empty hashref and nothing is recorded, so the same code works in
a cron job. `ForgeOps::Tracker::current_trace_id()` returns the current request's trace id, for your
own logs. To trace work that isn't a request but continues one (a job carrying the header it was
queued with, say), pass that header to `ForgeOps::Tracker::start_trace($traceparent)`.

A trace id exists for every request even with `track_tracing => 0`, and the header is still sent,
since the trace id is also what links an error here to an error in the service you called (see
[Where an error happened](#where-an-error-happened)); only span reporting stops. The service on the
other end must also report to ForgeOps, and both projects must be linked in ForgeOps to see their
errors and traces connected.

Narrow or turn off where the header goes, for example if a third-party API rejects unknown headers:

```perl
ForgeOps::Tracker::init(
    dsn              => '...',
    propagate_traces => 0, # never send traceparent (default 1)

    # Default undef: every host. A string matches that host and its subdomains on a dot boundary
    # ("internal.example" matches "orders.internal.example", not "notinternal.example"); a qr//
    # regular expression is matched against the host.
    trace_propagation_targets => ['internal.example', qr/\A10\.0\./],
);
```

### Where an error happened

An error reported during a request (from `Integrations::PSGI`/`Integrations::Dancer2`, or your own
`report` call from inside a route) carries up to three extra fields:

- `trace_id`: the request's W3C trace id, which ForgeOps uses to link this error to errors that
  other services reported for the same trace.
- `transaction_name`: the same name performance monitoring and traces use, `"GET /users/:id"`.
- `endpoint`: the HTTP method and the route as declared, `"GET /users/:id"`. Never the literal
  path, so an id or a token in the URL never ends up here.

`Dancer2Performance` fills in all three as soon as Dancer2 has matched the route, before the route
itself runs. `PSGIPerformance` fills in only `trace_id`, since plain PSGI has no route pattern (a
raw path is neither a transaction name nor an endpoint); if your PSGI app has its own router, name
the request yourself with `ForgeOps::Tracker::set_request_route('GET /users/:id', 'GET /users/:id')`.
They need one of the performance integrations enabled, are left out entirely outside a request, and
are never PII-scrubbed: they're structured fields, not free text. An error that escapes the request
keeps them (and the affected user and breadcrumb trail) even though `Integrations::PSGI` sits outside
`PSGIPerformance` and reports it after the request's own state has been cleared.

## Custom metrics and infrastructure monitoring

Two explicit calls (nothing is automatic, so there is no `track_metrics` option): a business event you
name yourself, and a reading from one of your own hosts.

```perl
ForgeOps::Tracker::capture_metric('signup');          # value defaults to 1: a bare counter
ForgeOps::Tracker::capture_metric('payment', 49);     # a real magnitude; it may be negative (a refund)

ForgeOps::Tracker::capture_infrastructure_metric('cpu', 0.42);                    # hostname defaults to server_name
ForgeOps::Tracker::capture_infrastructure_metric('disk', 0.81, hostname => 'db-1');
ForgeOps::Tracker::flush_metrics();                                                # optional: send right now
```

Each capture is buffered and flushed as one batch every `metric_flush_interval` /
`infrastructure_metric_flush_interval` seconds (60 by default) on a background thread, and once more
from an `END` block when the program ends normally, so a short-lived cron script that captures a few
readings and falls off the end needs nothing more (a test runs exactly that in a child `perl`); call
`flush_metrics()` if it might exit another way (`POSIX::_exit`). Every entry is stored as it was
captured (a signup is a row, not a running total), so a count or sum you compute later is exact. Both
are a no-op when the client isn't enabled for the environment.

A delivery that fails for now (a network error, a timeout, a 5xx, a 408 or a 429) keeps every entry
for the next flush, and an entry captured while a delivery is in flight is kept too (a test pins this
with a gated delivery on a second thread). A batch the server rejects (any other 4xx, such as the 403
a plan without the feature answers) is dropped rather than sent again, and so is one that has failed
5 flushes in a row. The buffer holds at most 1000 entries per kind and drops further ones until a
flush lets go of them. A NaN, infinite or non-numeric value is dropped at capture. Requires a ForgeOps
plan that includes custom metrics / infrastructure monitoring.

## What changed

ForgeOps can show what changed in your system next to the errors and slowdowns that followed it.
Two ways in:

**Record a change yourself** when something changes that no deploy captures, like a feature flag
flipped, a config value edited, or a migration run by hand:

```perl
ForgeOps::Tracker::record_change(
    kind    => 'feature_flag',   # feature_flag, config, migration, dependency, infrastructure, or other
    title   => 'Enabled new_checkout for 10% of users',
    details => { flag => 'new_checkout', rollout_percent => 10 },
    actor   => 'ops@example.com',
    url     => 'https://flags.example.com/new_checkout',
);
```

`kind` and `title` are required; `details` (a hashref), `environment` (defaults to the configured
one), `service`, `actor`, `url`, `id` (an idempotency key, so sending the same change twice records
it once), and `occurred_at` (an ISO 8601 string or epoch seconds, defaulting to now) are optional.
An unknown `kind` is sent as `other`. It's queued for the same kind of background thread as error
events, so it never slows down the caller, never dies, and does nothing when the client isn't
enabled for the environment.