App-karr
view release on metacpan or search on metacpan
ex/README.md view on Meta::CPAN
from the root is a good sanity check.
- `git` on `PATH`.
- Nothing else: no server, no network, no credentials.
## Build the sandbox
```bash
./ex/setup.sh # create (idempotent: existing pieces are left alone)
./ex/setup.sh --reset # wipe the generated state and rebuild from scratch
```
The script creates `ex/webapp/` and `ex/docs-site/` (each a git repo with a
karr board seeded with sample cards and a `.karr` file), `ex/fleet-hub/` (the
fleet's hub repository holding the chain), `ex/config.yml` (the
karr-foundation config pointing at all of them), and writes a demo chain into
the hub.
By default the sandbox calls the `karr`/`karr-foundation` binaries from this
checkout (`perl -Ilib bin/karr …`). Set `KARR_BIN` to point at an installed
binary if you would rather exercise that: `KARR_BIN="$(command -v karr)"
./ex/setup.sh`.
## Look at the boards
```bash
perl -Ilib bin/karr --dir ex/webapp board
perl -Ilib bin/karr --dir ex/docs-site board
```
`webapp` carries six cards — a login bug being worked, a rate-limit card that
depends on it, release notes in review, a blocked TLS upgrade, integration
tests, and a cross-board release card waiting on `docs-site#1`. `docs-site`
carries two.
## Run the fleet
A karr-foundation run is one pass over every repository in the config:
```bash
perl -Ilib bin/karr-foundation --config ex/config.yml --status # read-only overview
perl -Ilib bin/karr-foundation --config ex/config.yml --dry_run # what would run
perl -Ilib bin/karr-foundation --config ex/config.yml # the run
```
What you see on a fresh sandbox:
```
TICKET task#5
START agent=demo command=.../fake-agent.sh
fake-agent: working on #5
Moved task 5: backlog -> review
Handed off task 5 -> review (claim released)
START command=.../drain-agent.sh
drain-agent: working on #1
Moved task 1: backlog -> done
drain-agent: nothing to pick
```
- **webapp runs in `mode: ticket`.** karr-foundation picks the next assignable
card itself and tells the agent about it (`$KARR_TASK`, plus a line in the
prompt). `fake-agent.sh` moves it to review and hands it off under a fresh
claim, releasing the claim at the end.
- **docs-site runs in `mode: drain`.** The agent picks its own work
(`drain-agent.sh` uses `karr pick`). It works each card all the way to
`done`, because a drain run only terminates when no actionable cards are
left — an agent that parks cards in review would keep the drain going.
Run it again and watch `Moved task 5: review -> review`: a card in review is
still *actionable*, so it keeps getting picked until somebody moves it to a
terminal state. Finish it yourself with:
```bash
perl -Ilib bin/karr --dir ex/webapp move 5 done
```
## Scenarios worth trying
**A stalled agent and auto-block.** Point `docs-site`'s `.karr` at
`lazy-agent.sh` (which prints "I cannot make progress" and exits 0). A run
classifies as *stall* — the agent was given the board but nothing moved. After
`max_attempts` (2 in the sample), the card it keeps failing on is
auto-blocked. Reset the board with `--reset` afterwards.
**A failing agent and cooldown.** Point a `.karr` at `failing-agent.sh` (exits
1 with an "API error"). The run is a *common-error* and the repository goes
into cooldown, starting at 1m and backing off exponentially — visible in
`--status` as the repo being on hold.
**The chain.** The hub already holds a demo chain (`scripts/write-chain.pl`
wrote it): a docs build, a smoke test, a *question* step ("Shall we publish
0.6?") that gates a publish step. Execute what is ready:
```bash
perl -Ilib bin/karr-foundation --config ex/config.yml chain
```
On a fresh sandbox the shell steps run (docs, smoke) and the question step
goes *stale*: nothing in the question mailbox names it, and a question step
does not ask its own question — the planner asks it:
```bash
perl -Ilib bin/karr-foundation ask "Shall we publish 0.6?" \
--config ex/config.yml --step registry --options yes,no \
--policy use_default --default yes --wait 3600 \
--context "smoke and docs are green"
perl -Ilib bin/karr-foundation answer 1 yes --config ex/config.yml \
--note "all green, ship it"
```
A step that went stale stays stale — re-plan the chain so the answered
question resolves it:
```bash
perl -Ilib ex/scripts/write-chain.pl ex/fleet-hub
perl -Ilib bin/karr-foundation --config ex/config.yml chain
# step registry (question): done — question #1 answered 'yes'
# step publish (shell) in .../webapp: done — exit=0
```
## How the pieces fit
ex/README.md view on Meta::CPAN
**Whether an agent currently works is a property of the machine and the
account.** A spent quota belongs to a person, not a project. If it were board
state, a push would distribute the spent limit of *one* person to the fleet of
*all*.
**How many agents may run at once is a property of the machine.** It protects
the CPU and memory of this box. It says nothing about what any account may
spend.
Hence: `.karr` and `config.yml` and `agents.state` are local, always, without
exception. The one thing that belongs to execution and is still synchronized is
the refusal — `karr disable` — and that is deliberately board state, because
"nothing shall run automatically on this board" is a statement about the
*project* and not about the machine.
### The first run
With nothing configured at all:
```console
$ karr-foundation
karr-foundation: config not found at /home/dev/.config/karr-foundation/config.yml — nothing to do
karr-foundation: no repos found — check config
$ echo $?
1
```
Naming the repository:
```yaml
# ~/.config/karr-foundation/config.yml
dirs:
- /srv/webapp
```
That's already enough for the read-only half:
```console
$ karr-foundation --status
webapp
3 tasks
backlog:1 todo:2
```
An ordinary tick still does nothing, and says why:
```console
$ karr-foundation
No agent will run on any board. Showing overview (set 'command:', 'agent:' or 'claude: true' in a .karr file to enable agents; a board disabled with 'karr disable' never runs one).
webapp
3 tasks
backlog:1 todo:2
```
Now the opt-in, and that is **in the repository itself**:
```yaml
# /srv/webapp/.karr
mode: ticket
command: my-agent --task "$KARR_TASK" --prompt "$PROMPT"
max_runtime: 1800
max_attempts: 2
```
### The three modes: `drain`, `single`, `ticket`
`mode:` says what **one pass over a repository** is:
| Mode | Meaning |
|---|---|
| `drain` (default) | run the agent again and again until the board stops moving |
| `single` | exactly one agent run; the agent finds its own work |
| `ticket` | exactly one agent run, over **one card that foundation names** |
`drain: true|false` is the older spelling of the first two and stays valid
(`true` = `drain`, `false` = `single`). Two keys that both mean "one run" would
be a trap, so it's one key with an alias: `mode` is asked first, `drain` only
answers when `mode` is missing, and a `drain` per repository still wins over a
`mode` from the global configuration. An unknown `mode` is an error that skips
the repository — never a silent fallback to "well then drain".
**Ticket mode deserves its own explanation**, because it's the mode the chain
builds on. Before starting the agent, foundation selects the card that the run
is about — with `karr pick`'s own eligibility and ranking check (not terminal,
not blocked, not held by a live claim; class, then priority, then identifier).
These rules live in *one* role that both `karr pick` and foundation's picker
compose. Until that was the case, they stood written out twice, and only the
fact that they were copied from each other kept them in agreement — while a
coordinator that names a different card than the board would hand out argues
with its own board.
The card is told to the agent **twice**: as the closing sentence in the
`$PROMPT`, which names the identifier, and as `$KARR_TASK` for a command
template that wants the bare number. That the identifier is spliced into the
prompt and not the prompt itself writes `$KARR_TASK` has a prosaic reason: the
prompt reaches the agent as `$PROMPT`, and `/bin/sh` doesn't rescan an expanded
value — a prompt that contained `$KARR_TASK` would give the agent those ten
characters.
And: **foundation names the card, it doesn't claim it.** The claim is the
agent's work session, minted with `karr agentname` and reused across its own
`move` and `handoff` calls. A claim invented by foundation couldn't be handed to
the agent without a protocol of its own. The board's `.karr.lock` and the rule
"one agent per repository" keep everyone else away for the duration of the run
anyway. An agent that dies mid-run leaves at most its own claim — resolved by
`claim_timeout` or `karr unlock` — and costs one attempt on foundation's
counter.
If there is no assignable card at all, **no agent** runs in ticket mode;
`.karr.log` gets `TICKET none assignable`, and the result is `idle`. `--force`
and `on_idle: always-run` force the check, not a run without a card.
### Look first, then run
```console
$ karr-foundation --dry-run --verbose
sync --pull /srv/webapp
[2026-08-18T05:35:34] 1565842: TICKET task#1
[2026-08-18T05:35:34] 1565842: START command=my-agent --task "$KARR_TASK" --prompt "$PROMPT"
exec in /srv/webapp: my-agent --task "$KARR_TASK" --prompt "$PROMPT"
[2026-08-18T05:35:34] 1565842: DRY-RUN (skipped)
[2026-08-18T05:35:34] 1565842: STALL task#1 — no report from the agent
```
A dry run starts nothing and writes nothing — no agent, no `.karr.state`, no
`.karr.log`, and not even the pull that the first line announces. It's also mute
without `--verbose`: those lines are the verbose stream, not a report.
Then the real thing:
```console
$ karr-foundation --verbose
sync --pull /srv/webapp
[2026-08-18T05:35:34] 1565844: TICKET task#1
[2026-08-18T05:35:34] 1565844: START command=my-agent --task "$KARR_TASK" --prompt "$PROMPT"
exec in /srv/webapp: my-agent --task "$KARR_TASK" --prompt "$PROMPT"
working on #1 as fund-duty
[2026-08-18T05:35:35] 1565844: END elapsed=1s exit=0
```
The agent's output is streamed to the terminal in real time when there is one
(or with `--verbose`), and always appended to `.karr.log`. The card has moved,
so there's nothing more to say. `.karr.state` now carries the board fingerprint
against which the next tick compares:
```json
{"hash":"e256a65404833f0e801b323e1e2301fd","last_exit":0,"last_run":"2026-08-18T05:35:35"}
```
The command is a **shell template**, not a string that `karr` rewrites. Exported
into the child's environment: `PROMPT` (the instruction), `KARR_REPO` (where it
is), `KARR_ROLE` (`agent` or `hook`) and `KARR_TASK` (the identifier in ticket
mode, otherwise empty).
### How a run is judged
After every agent run, foundation classifies the result from what it can observe
— the run's own report where the agent gave one, otherwise from the exit code,
the board's ref movement and the captured output:
| Result | What it means | What follows |
|---|---|---|
| **progress** | the board has moved (in ticket mode: *this* card has moved) | keep draining |
| **stall** | a card the agent was working on hasn't moved | increment that card's attempt counter; at `max_attempts` automatically block |
| **common-error** | non-zero exit, timeout, or an error pattern in a run that moved nothing | no card is punished; the repository goes into exponential cooldown |
| **idle** | the agent did nothing and took nothing | stop |
The order of evidence is intentional: **what the run did is asked before what it
printed.** A run that ended with 0 and moved the board is progress, no matter
what text scrolled by, and is never reclassified by its own transcript. The text
scan is evidence only where there is no other — for a run that produced no board
movement at all, which looks exactly like a rate-limited or unauthenticated
agent. A pattern in a run that moved the board **after all** is noted in
`.karr.log` and otherwise ignored.
Correspondingly tight are the default patterns: a symptom word only counts next
to an error word in the same line ("network error", "invalid credentials",
"quota exceeded"), not on its own, and an HTTP status only counts where
something next to it identifies it as such ("API error: 429", "429 Too Many
Requests") — not in a diffstat and not in a line number. Before that, an agent
that printed its own board triggered the scan on a backlog title, and a diffstat
with 403 changed lines on the `403`. `error_patterns:` in the `.karr` adds your
own case-insensitive substrings.
**The run's own report.** An agent invoked with `--output-format json` ends its
output with one line: a JSON object that says whether the run failed, how it
ended, how many turns it needed, how long it ran and what it cost. Where there
is such a report, foundation classifies from it, and the text scan doesn't run
at all.
Foundation isn't **configured** for it and doesn't inspect the command for it
either — it reads the end of the output, because that's where the format puts
its result, and then nothing has to be kept in agreement. Only the **last**
non-empty line counts: prose before it is irrelevant, prose that itself contains
such an object can't be confused with it, and anything after it makes the run
unstructured again, upon which the scan takes over.
ex/README.md view on Meta::CPAN
The unit of concurrency is a forked child process that runs the whole pass for
exactly one repository and holds its `.karr.lock` for the duration of its drain.
Forking the whole pass instead of teaching the drain loop to interleave several
agents is the cheap half of this rule: every piece of board state — the lock
file descriptors, `.karr.state`, the engagement record, the attempt counters —
keeps exactly one writer, without a single line of it changing. The only truly
shared state is `agents.state`, and that's locked.
Three levels bound what actually runs, and **the tightest wins**:
| Level | Where | What it is |
|---|---|---|
| machine cap | `concurrent:` in the local configuration | protects the CPU and memory of this box; default `1` |
| operator estimate | `concurrent:` on an agent definition | roughly where this agent's session limit sits |
| plan announcement | `limits:` in the chain header | what this one run declares; can only tighten |
The middle number is an **estimate about someone else's rate limit** and is
allowed to be wrong, because being wrong is cheap here: the agent starts
failing, is marked as such, every board on it is skipped for a probe interval,
and the fallback takes over. This number needs no more error budget than that.
A broken number is treated by where it comes from. In the local configuration
it's a user error — it's the operator's own file, and a `concurrent: "two"`
that silently means one is exactly the quietly wrong answer this distribution
rejects. In the chain header it warns and is ignored: the header was written on
another machine, and not running the fleet because of a foreign typo is worse
than running it at the local cap. Likewise, a `per_agent:` name that this
machine doesn't define is dropped with a verbose note instead of rejected —
agent definitions are local and only local, so a chain written on a machine with
`minimax` that reaches a machine without `minimax` is the normal case and not a
broken plan.
The scheduler **searches** the queue instead of working through it in order: a
board whose agent sits at its own limit is passed over, not waited out.
Head-of-line blocking here would let one busy agent idle the whole machine while
boards wait on another agent.
`--dry-run` stays serial no matter what the cap says: it starts no agent, so
concurrency would buy nothing and would cost the output the order you read it
in.
**`hub:`** names the one repository of a fleet that carries
`refs/karr-foundation/*` — the chain, its run logs and the questions mailbox.
That's a **role, not a separate project**: any repository can take it over, and
a fleet without a dedicated coordination repository appoints one of its own to
it. The local configuration names it. An ordinary tick pulls this namespace
before it reads anything and writes nothing back; executing the chain is a
command of its own.
A missing hub is a **warning, not an error** — everything foundation does
without the chain, it keeps doing. Only the commands that mean nothing without a
hub (`ask`, `answer`, `chain`) fail hard.
### `on_drained` — the hook that `karr` deliberately doesn't understand
When a board has been drained — no workable card left, everything done, archived
or blocked — foundation may run exactly one command in it:
```yaml
# /srv/gate/.karr
command: my-agent --prompt "$PROMPT"
on_drained: ./release-gate.sh
on_drained_max_runtime: 1800
on_drained_max_rounds: 3
```
```console
$ karr-foundation --verbose
sync --pull /srv/gate
[2026-08-18T05:35:54] 1566032: START command=my-agent --prompt "$PROMPT"
exec in /srv/gate: my-agent --prompt "$PROMPT"
no card assigned, nothing to do
[2026-08-18T05:35:54] 1566032: END elapsed=0s exit=0
[2026-08-18T05:35:54] 1566032: START role=hook command=./release-gate.sh
exec in /srv/gate: ./release-gate.sh
release gate in /srv/gate (role=hook)
[2026-08-18T05:35:54] 1566032: END elapsed=0s exit=0
[2026-08-18T05:35:54] 1566032: ON-DRAINED exit=0
```
**`karr` doesn't know what this command does, and must not know it.** In the
fleet this design comes from, it starts a release gate that builds a
distribution, installs it, tests every dependent consumer against it and raises
version requirements — across 44 distributions, of which `karr` may not learn a
single detail.
Everything domain-specific — what "done" means for a project, how a release is
verified, which project depends on which — reaches `karr` via `on_drained` and
via **nothing else**. That's why the exit code is written to `.karr.log` and
`.karr.state` and is **interpreted by nobody**: a failing hook doesn't park the
board, doesn't mark its agent as failing, and never becomes the run's
`last_error`. It isn't an agent run and isn't classified as one — no report is
read from it, no error pattern matched against it, no card assigned to it.
It's told where it is, and nothing else: `KARR_REPO`, and `KARR_ROLE=hook`, so
its own `karr` write operations land in their own activity log instead of
counting as an agent's tussle with a card. `PROMPT` and `KARR_TASK` are empty.
It runs in the board's directory, under its own `.karr.lock`, with the same
process-group kill and the same tee to `.karr.log` that an agent gets — a gate
that sends a build to the background must not outlive the run that started it —
but with its own budget (`on_drained_max_runtime`), because how long an agent
may take says nothing about how long a release gate may take.
**"Drained" is a fact about the board, not a name for a result**: there is no
workable card left on it. That's deliberately the same question that `--force`
and `on_idle: always-run` are answers to, and the only one that keeps the same
meaning across all modes. A drain that ends in a `common-error` doesn't count: a
rate-limited agent leaves behind a board that looks exactly like one it worked
through, and foundation doesn't believe that run itself.
**An empty board isn't the same as finished work.** The hook may fail and create
tickets; then the board is no longer empty, the next tick works them, the board
runs empty again, and the hook is asked again. This cycle **is** the point of
the thing — a gate that reports what it found and runs again as soon as it's
fixed. So it's bounded instead of forbidden, with two guards:
- **The same board isn't asked twice.** The board fingerprint at which the hook
last ran stands in `.karr.state`; a board that hasn't moved since doesn't get
a second run. Without that, a repository nobody touches would start a release
gate on every cron tick, forever — because an empty board stays empty.
- **A chain that never settles is capped.** Every hook run that puts work back
on the board changes the fingerprint, so the first guard can't see the loop
"hook creates a ticket, agent works it, board runs empty, hook creates the
next one". Consecutive rounds in which the hook itself produced work are
counted; a run that leaves the board alone — the gate that finally passed —
resets the counter, and at `on_drained_max_rounds` (default 3, `0` disables)
the hook is suppressed, with a line in `.karr.log` that says so.
`--force` overrides both. They're statements about board state, and `--force`
overrides exactly that, per its documentation; and unlike cooldown and agent
availability, the cap isn't bounded in time and doesn't end on its own — so it
ex/README.md view on Meta::CPAN
in-progress: #2
$ karr enable
Board enabled for automated agent runs (karr-foundation).
```
Unlike `.karr`, that's **board state** (`foundation.enabled` in
`refs/karr/config`), so it synchronizes, and every foundation instance on every
machine honors it. A disabled board is skipped **entirely**: the flag is checked
before the agent command is resolved and before the drain is decided — no drain,
no auto-block, no agent run. It wins over `--command`, `default_command`, the
`command` from the `.karr` and `claude: true`, and **`--force` doesn't override
it**. Disabled means disabled.
Otherwise nothing changes: the board stays fully usable for humans and for
hand-driven agents. The use case is a repository whose backlog is parked rather
than abandoned — a kept legacy project that a globally configured
`default_command` would otherwise drain through.
The same state is readable and writable via `karr config get` / `karr config
set` (`foundation.enabled`, `foundation.reason`). There's one truth in the board
configuration, and `disable`/`enable` are the ergonomic front door to it. `karr
enable` makes the `foundation` key disappear again entirely from the sparse
overrides, because the value is back on its code default.
### Reference
`karr-foundation` options:
| Option | Effect |
|---|---|
| `--config PATH` | configuration file (default `~/.config/karr-foundation/config.yml`); also moves `agents.state` |
| `--status` | read-only overview of every board, then exit |
| `--dry-run` | decide everything, execute nothing (serial, and mute without `--verbose`) |
| `--verbose` | log lines and agent output on the terminal, agent descriptions in `--status` |
| `--force` | run regardless of board state; with `answer`: replace an existing answer |
| `--command CMD` | one agent command for every board, overrides any `.karr` |
| `ask` / `answer` / `chain` | the hub commands (`--context`, `--options`, `--default`, `--policy`, `--wait`, `--step`; `--note`) |
**Exit codes** follow the same contract as `karr`:
| Code | Meaning |
|---|---|
| `0` | the tick ran through — boards drained, overview printed, question asked or answered, chain worked through |
| `1` | runtime error: no repository found, unreadable configuration, a hub command without a hub, an already answered question, or `chain` couldn't fetch `refs/karr-foundation/*` |
| `2` | usage error: unknown command, unknown option, invalid option value, missing or extra positional argument |
A **failed chain step doesn't change the exit code** — that's a statement about
the plan, not about the binary. A run terminated by `SIGTERM`, `SIGINT` or
`SIGHUP` ends with `128 + signal`, after taking its agents along.
`config.yml` keys:
| Key | Meaning |
|---|---|
| `dirs:` | explicit board repositories |
| `scan:` | parent directories whose direct children are checked for a board |
| `concurrent:` | machine cap of simultaneous boards with an agent (default 1) |
| `hub:` | the repository that carries `refs/karr-foundation/*` |
| `agents:` / `default_agent:` / `probe_every:` | named agent definitions, the fallback agent, the default retry interval |
| `default_command:` / `default_prompt:` | fleet-wide command and fleet-wide prompt |
| `mode:`, `claude:`, `claude_bin:`, `claude_max_turns:`, `claude_permission_mode:`, `on_drained:`, `on_drained_max_runtime:`, `on_drained_max_rounds:` | fleet-wide defaults for the `.karr` keys of the same name |
`.karr` keys per repository (each wins over the fleet-wide value):
| Key | Meaning |
|---|---|
| `command:` | the agent command; a shell template, `$PROMPT` and `$KARR_TASK` are exported into it |
| `prompt:` | the instruction passed as `$PROMPT` |
| `agent:` | a named agent from `agents:` |
| `claude:` / `claude_bin:` / `claude_max_turns:` / `claude_permission_mode:` | synthesizes the canonical claude command (opt-in) |
| `mode:` | `drain` (default), `single`, `ticket`; `drain: true\|false` is the older spelling of the first two |
| `on_idle:` | `skip` (default) or `always-run` |
| `max_runtime:` | SIGKILL per command in seconds (`0` = no timeout) |
| `max_attempts:` | stalls on a card until it is auto-blocked (default 2) |
| `max_iterations:` | hard cap for drain iterations (default 50) |
| `cooldown_base:` / `cooldown_max:` | cooldown minutes at level 0 (default 1) and the ceiling (default 64) |
| `error_patterns:` | additional case-insensitive substrings that count as common error |
| `on_drained:` / `on_drained_max_runtime:` / `on_drained_max_rounds:` | the domain hook, its budget and its round cap |
**Command resolution order**, highest first: `--command`, `default_command`, the
`command` from the `.karr`, the `agent` from the `.karr`, `default_agent`,
`claude: true`. A board disabled with `karr disable` runs none of them. A board
that names an agent the configuration doesn't define is an error that skips
**this board** — not one that silently stops everything.
Full details: `perldoc App::karr::Foundation`, and for the chain
`perldoc App::karr::Foundation::Executor`,
`perldoc App::karr::Foundation::ChainStore`,
`perldoc App::karr::Foundation::Questions`,
`perldoc App::karr::Foundation::Agents`.
---
## 5. The architecture idea
Everything up to here is mechanics. The reason the mechanics look the way they
do and not otherwise is three layers.
### The three layers
**Coordination — shared, in refs.** Cards, dependencies, escalations, the chain
of planned steps, open questions, the run log. All of it lies in Git refs and is
synchronized with the remote, so every machine and every person sees the same
picture.
**Execution — local, never in the repository.** Which agent commands exist on
this machine, whether they currently work, when to try again, how many may run
at once. Never in the repository: an agent command that exists on one machine
may not exist on the next, and an account limit is a property of a person, not
of a project.
**Judgement — an agent.** Planning, routing, reacting to the unexpected. That's
a coordination agent, invoked like any other agent, and **only when a written
plan is missing or broken.**
`karr` owns the first two. The third is the one that was never built — see
section 6.
### The sentence that carries the whole thing
> **The AI is the compiler, the chain is the program, karr-foundation is the
> VM. No agent runs in the hot path.**
That's the interesting part, and it's worth unpacking.
The obvious construction for "coordinate several agents across several
repositories" is an orchestrator agent: a language model that reads the state of
all boards, decides what happens next, and then starts workers. That works
( run in 0.358 second using v1.01-cache-2.11-cpan-aadc1410aed )