The worker contract

How a spawned worker's run is allowed to end, and who says so.

cli/lib/spawn.js has cited this file since #70; it is only now being written down. It is the doc of record for the endings — if you change how a run terminates, change it here first.

The problem it exists to solve

A worker used to end its life at an idle prompt: task finished, nothing said, process still up. Three workers forgot to report in one day. A contract an agent has to remember is not a mechanism, so both halves — REPORT and DIE — were moved into the harness:

• Reporting is mechanical. A headless worker runs under claude -p --output-format json, so its final message is its report; crux supervise lifts it onto the thread. The agent does not choose to report, and cannot forget to.
• Dying is mechanical. -p exits when the work is done. Its exit is the reap.

The load-bearing consequence: silence is the bug. A process that ends having printed nothing has told the thread nothing, and the thread would wait on it forever. So the supervisor speaks for it.

The endings

A run ends in exactly one of these. The supervisor decides which, and posts it to the thread. (One outcome is not in the table because nobody is left to report it — the supervisor died too and nobody said anything — and it is below.)

Reported is the happy path. The agent's own words go on the thread with a provenance footer (turns, cost, claude --resume <session>). "Done!" is technically a report and tells you nothing — the brief says so to the worker's face.

Died covers every way a run can end without a usable report, because from the thread they all used to look identical: nothing. No json printed; is_error; a non-zero exit (Solo surfaces it as failed); an empty result; or a run that outlived the harness's patience. A dead worker also pushes, because a note nobody is told about is just a quieter silence.

Report unreadable is the newest of these, and it exists because "no json parsed" was read as "no json printed" for as long as the two shared a message. resultJson anchored on {"type":"result" — type as the object's FIRST key — and claude -p --output-format json moved it to second to last. Three workers that had each committed, pushed and opened a pull request were announced as crashes and pushed to a phone in one night (#357). So the supervisor now asks the buffer a narrower question: did a result object go out at all? An unescaped "type":"result" says yes, whatever else is missing. When the answer is yes, crux recovers what it can — the whole report if only the run's bookkeeping was lost, the report's tail if not — and says on the thread that it is partial. It is not a crash and does not push, because the work is fine and the failure is crux's own. A buffer with no result object in it is still a corpse, and is still announced as loudly as ever.

Cancelled by lead is the third ending, and it exists because of a real incident (thread #74). A lead spawned a worker, the owner changed the approach mid-flight, and the lead stopped the worker with solo processes stop. The supervisor saw a process that ended having printed nothing — which is exactly what a crash looks like — and posted "Worker failed … it crashed, was killed, or never started", plus a push to the owner's phone. Nothing had failed. Every mid-flight re-brief did this, and a crash note that cries wolf is worse than no crash note, because the real ones start being ignored.

How a cancellation is told apart from a crash

Not by observation. It cannot be: process gone and buffer empty is the same event either way. The supervisor is not made smarter — the killer is made to leave a note.

crux cancel <id> --note "why" --actor lead-crux

1. Writes a tombstone, keyed by item id AND Solo process id, under ~/.config/crux/cancelled.json (same lock + write-then-rename as the tail cursor — see cli/lib/store.js).
2. Then stops the Solo process.
3. Then posts a plain worker cancelled by lead: <why> note to the thread. No push: a lead stopping its own worker is not an escalation.

The supervisor, already awake and polling, sees the process end, finds the tombstone, consumes it, and reports a cancellation instead of a death.

Four rules keep that from becoming a way to hide real crashes:

• Order. Tombstone before the kill. Written after, it races a supervisor that polls every 5s and loses often enough to report a crash anyway. A cancel whose tombstone cannot be written aborts before killing anything, rather than killing the worker and then lying about why it died.
• Key. <item>:<process>, never <item>. Thread #74 was cancelled and re-spawned onto a new process minutes later — an item-keyed tombstone would have been lying in wait to excuse the replacement worker's genuine crash.
• Default. No tombstone still means crash, loudly, with a push. This is the #70 guarantee and it does not move. Silence that nobody claims is precisely the bug the supervisor exists to catch.
• Once. The tombstone is deleted under the same lock that reads it, and one older than 24h is ignored on sight and swept up. The worst a stale or orphaned tombstone can do is nothing.

A tombstone only excuses the deaths a kill actually explains. A timed-out run is still running (the kill never landed), and a worker that got its report out before the axe fell has reported. Neither is rewritten as a cancellation.

The fourth ending: nobody said anything at all

Everything above is decided by crux supervise — a local pid sitting beside the worker. That is the flaw, and it is not a small one: anything that takes out both processes at once (a laptop closing, a reboot, a Solo restart, the OOM killer) takes out the very thing whose job is to say the worker died.

It happened on #229. Worker process 427 was spawned, did 166 lines of real work, and exited having posted nothing. Its supervisor was gone too. The thread sat in working for 22 hours flying a status line that read "worker on it", which is indistinguishable from a worker that is still working. Nobody was paged. The uncommitted diff in the worktree was the only evidence it had ever run.

A witness that shares fate with what it watches is not a witness. So there is a second one, and it runs on the server:

php artisan crux:reconcile-workers          # scheduled every 15 minutes
php artisan crux:reconcile-workers --dry-run  # what it would say, writing nothing

It looks for threads that are working, have a spawned agent_process_id, and have had nothing posted to them by the worker since the moment they were accepted. When it finds one:

Reap is the half that matters. Announcing alone corrects the caption and leaves state reading working — the exact lie that hid #229 for 22 hours. It fixes the caption and leaves the photo.

All three settings belong to the ACCOUNT, not the server (#321). They shipped as env vars, which meant the answer one operator gave about his own phone was applied to all sixteen tenants: other people's threads reaped and other people's phones buzzed, on a schedule, from a preference they never expressed. Announce is the only one nobody opts into — it is the honest minimum, and the only one of the three that neither changes your data nor wakes you at 3am. Change your own:

curl -sH "Authorization: Bearer $CRUX_API_TOKEN" $CRUX_API_URL/api/account/settings
curl -X PATCH -H "Authorization: Bearer $CRUX_API_TOKEN" -H 'Content-Type: application/json' \
  -d '{"worker_silence":{"minutes":240,"reap":true,"notify":true}}' \
  $CRUX_API_URL/api/account/settings

CRUX_WORKER_SILENCE_MINUTES / _REAP / _NOTIFY still exist and now mean exactly one thing: what a newly created account is born with. They cannot reach an account that already exists.

Four hours by default, and the number is measured rather than picked — on one person's traffic. Across all 248 acceptances in the live database (2026-07-31) the gap from accepted to the next activity ran p90 24 minutes, p99 119 minutes, and max 176 minutes, so 240 is about 1.4× the worst healthy silence ever observed and would have produced zero false positives against everything that has ever happened there. Those acceptances were overwhelmingly Daniel's; somebody working a different rhythm — slower reviews, a laptop that sleeps, another timezone — gets a different false-positive rate, which is why it is a number each account can move (15 minutes to 7 days).

Four things keep it from becoming #266 at a larger scale — a detector that cries wolf gets the real ones ignored, and this one runs unattended on every thread at once:

• Proof of life is the WORKER speaking (#319). A lead's note or your own reply does not reset the clock: on #230 the worker died at 00:24, the lead posted at 01:45, and the reconciler reported "silent 18m" about a corpse that was 101 minutes cold. Everything that is definitionally somebody else — the human, a lead, the router, a supervisor, the reconciler itself — is discounted.
• Only since the accept. A reused thread's older chatter cannot vouch for a new worker.
• Only a spawned worker. No agent_process_id, no verdict — somebody working a thread by hand in a terminal is not something a process death can happen to.
• It says "silent", never "dead". A worker that is alive and merely quiet looks identical from the server, and the honest word for what it can actually see is the one it uses.

A thread already flagged is not flagged again on the next tick; the flag clears only when somebody is genuinely heard from.

What happens after a reap. Nothing auto-spawns. Reaping drops the thread to idle (or acknowledged, if a lead had filed it), which puts it back in front of the project lead — a live agent parked on crux tail, which wakes on the silent activity and decides. There is no server-side spawner at all, and the router explicitly does not spawn. Should a lead decide to respawn, crux spawn refuses if that thread already has a live Solo worker (#311), so a false positive cannot put two agents in one worktree.

cancel vs done

• crux cancel <id> — the brief is now wrong. Stops the worker; the thread stays open, ready to be re-briefed and spawned again.
• crux done <id> — the thread is finished. It closes, and its worker dies with it (an agent's life is bounded by its thread's).

What the worker itself is told

Two endings, not three — cancellation is not something an agent chooses, it is something done to it. Every brief is wrapped with:

1. DONE → your final message IS your report; it is lifted onto the thread the moment you exit.
2. BLOCKED → crux need --item <id> --kind decide|answer|secret|desk|world "<question>". It is the only thing that reaches your phone.

Plus, for a headless worker, the thing that is true and easy to forget: there is no prompt to wait at, and nobody is watching this process. Waiting silently is not patience, it is death.

…and how to write it

The report is published, not filed: it lands in a chat bubble and is read on a phone. So the DONE clause says what a report looks like, not only that one is required — never name a PR, issue or thread without its URL (gh pr view <n> --json url -q .url), bold what matters, fence commands and output, answer first and detail after.

This is the same rule the generated lead brief carries under How you write (renderLeadBrief in cli/lib/swarm.js), and it is stated in both places on purpose. A lead writes the task half of a worker's brief; crux spawn writes the wrapper. A worker whose lead never heard the rule still gets it from the wrapper, and a report the reader has to go and reconstruct — a bare #29, a wall of prose — is crux failing at its one job, silently, on a thread that looks answered.