crux swarm — standing leads for your projects
Run agents that work your projects while you are not there, and interrupt you only when the call is yours.
You need two things and nothing else: the crux CLI on your PATH, and a repo on disk. crux install
points the CLI at the server and your account, crux discover turns the repo into a crux project, and
the section after that gets a lead running on it in about five minutes.
Install the CLI
npm install -g @coulb/crux-cli
Node 18+, zero dependencies. It installs one command, crux — everything on this page is that
command. Check it landed:
$ crux --help
Step one: crux install
npm install -g put a command on your disk. It did not tell that command which server it talks to, or
who you are. crux install is the one that does, and until it has run once, every other verb on
this page can only tell you that it has no token.
$ crux install
crux install — writes ~/.config/crux/config.json, which is what every other crux command reads.
Server URL [https://crux.coulb.com]
Your API token — the one thing crux cannot work out for itself.
Don't have one? Sign up at https://crux.coulb.com/register
The token is shown ONCE, on the page you land on right after you
register, under "Your API token — shown once". Copy it there.
It is not your password, and there is no second chance to read it.
API token (not echoed):
Checking https://crux.coulb.com… ✓ the token works (0 projects).
✓ ~/.config/crux/config.json — created.
url https://crux.coulb.com
token verified, and never printed
mode 0600 — this file is a bearer token to your whole feed
Next: crux discover # turn a repo into a crux project
Two questions, and it only really asks one of them. The URL is defaulted: https://crux.coulb.com
is the answer for everyone who is not self-hosting, so Enter takes it (--url is there for everyone
else, and a bare host means https). The token is asked, because it is the one thing only you
have.
Where the token comes from
Sign up at crux.coulb.com/register. The token appears once, on the page you land on immediately after registering, in a green banner headed "Your API token — shown once". It is stored hashed, so that render is genuinely the only time it will ever exist on a screen. Copy it then.
It is not your password. A crux email and password will not authenticate the CLI, and pasting one at the token prompt gets you a 401.
If you lose it, there is no way to reissue a CLI token today — no list, no revoke, no second look. (The phone app's sign-in,
POST /api/login, does mint a fresh one from your email and password; acrux loginverb that uses it is filed, not built.)
What it will not do
| write a config that does not work | The token is checked against the server (GET /api/projects) before the file is written. A 401 fails the command and writes nothing — a config that is saved and then does not work is worse than no config, because you now believe you are set up and every later error is about something else. |
| clobber the one you have | An existing config is shown — the URL, never the token — and you are asked. --force means it. Under --yes there is nobody to ask, so it refuses instead: the token in that file was shown to you once and cannot be shown again. |
| echo your token | Not to the terminal (the prompt does not echo), not into your shell history, not into --json, not into an error message. |
| delete anything | Not even a config it cannot read. See below. |
The file lands 0600. It holds a bearer token to your entire feed.
The config that is already there and is not a config
Could not parse your config. — if every crux command is saying that, the file at
~/.config/crux/config.json exists and is not JSON, and you are stuck: the command that writes that
file is a crux command too.
crux install gets you out of it. It moves the unreadable file aside — to config.json.broken, and it
tells you where — and writes a working one. Nothing is deleted: those bytes may be the only copy of
a token that was shown to you once, and crux is not the judge of that.
$ crux install
~/.config/crux/config.json exists, but it is not valid JSON — crux cannot read it, and neither can
any other crux command, which is probably why you are here.
crux can move it to ~/.config/crux/config.json.broken and write a fresh one. Nothing is deleted — if
there is a token in there, it will still be there, in that file.
Move it aside and write a new config? (y/n) [y]
In a script, where nobody can be asked, it refuses and names the flag that means it: crux install --force. And the move happens after your new token has verified — a run that cannot authenticate
leaves the file exactly where it found it.
An empty file (touch, an interrupted write) is not a config at all, and is simply written over.
So is a config holding neither a url nor a token: there is no credential in it to protect.
crux install --url http://localhost:8000 # self-hosting
CRUX_API_TOKEN=crux_… crux install --yes # a script, or a machine being provisioned
crux install --force # replace the config already there
crux init # the same command, under the name you may reach for
CRUX_API_URL / CRUX_API_TOKEN in the environment still win over the file — that precedence is
unchanged, and crux install says so out loud if one of them would shadow what it just wrote.
CRUX_CONFIG puts the file somewhere else entirely.
Step zero: crux discover
A lead works a project, in a repo. crux discover is how a repo becomes both — point it at a
directory, answer a few questions, and you have a crux project registered with the API and a repo
path on this machine for its lead to stand in. It is the first command you run about your code, and
it is the one this whole page assumes you already ran.
$ cd ~/src/acme
$ crux discover
Found ~/src/acme — a Laravel app.
slug [acme]
name [Acme]
color (emerald violet sky amber rose pink zinc) [emerald]
keywords [acme,laravel,php]
repo [/Users/you/src/acme]
✓ acme is a crux project — created.
repo ~/src/acme
config ~/.config/crux/leads.json — swarm.projects.acme.repo, swarm.router.repo, swarm.backend,
swarm.exec added
lead not started
runs on exec (auto-detected, tmux template) — written to the config above, edit it freely
Also wrote swarm.router.repo — the working directory for the router that reads your
UNROUTED queue (everything you capture that matches no project's keywords). It has no
code of its own and will not touch this repo; `crux swarm up` starts it alongside your leads.
Next: crux swarm up --project acme
That last one is the router, and it is worth reading about before you get far — it is what keeps a captured thought that matches none of your keywords from sitting in a queue nobody is watching.
Enter, Enter, Enter. It does not ask what it can read. The repo path comes from git (run it in a
subdirectory and it still finds the root). The name comes from your origin remote, then
package.json / composer.json, then the directory. What the project is — Laravel, a Node CLI,
Rails, Go — is read from the manifests, and it matters twice: a lead that knows it is standing in a
Laravel app writes better briefs, and the framework becomes a routing keyword.
Those keywords are the part worth reading before you press Enter. They are how a thought typed into your phone at midnight ("the acme importer keeps timing out") gets routed to this project instead of landing in the unrouted queue. Derived from every name the project already answers to; edit them if they are wrong.
--yes takes every default and asks nothing, which is the same run with the Enters implied:
$ crux discover ~/src/acme --yes
It writes two places, and needs both
| what | why | |
|---|---|---|
| the API | POST /api/projects |
the project exists in the feed, the app and the web |
| this machine | swarm.projects.acme.repo in the config |
crux swarm up knows which repo the lead runs in |
A project with no repo cannot be led, and a repo with no project has nothing to lead — so it does both or it does neither. If the config is not writable, it says so and creates nothing, rather than leaving you a project in the feed that no lead can ever be started for.
And it settles what will RUN the lead
runs on is the third thing discover writes, and it is there because of a first run that used to end
like this:
$ crux discover ~/src/acme --yes
Next: crux swarm up --project acme # ← discover said this
$ crux swarm up --project acme
crux: the exec backend needs a command template — set `swarm.exec.command`, …
A step-zero command that prints a step-two command it never checked. So discover now looks at your machine — Solo, then tmux — picks one, and writes the choice into your config where you can read it and change it. Derived, never asked: the PATH already knows the answer, and being asked a question your computer could have answered is what makes a tool feel like a form.
If it finds nothing, it says so there — with the one line to add and the file to add it to — and
prints no Next: at all. It will not hand you a command it can already see will fail.
✓ acme is a crux project — created.
runs on nothing yet — see below
✗ crux found no orchestrator on this machine — a lead is a long-running agent, so
something has to host it.
Install one, and crux will find it — no configuration:
brew install tmux # the simplest: each lead becomes a tmux session
Or bring your own, by naming the command that starts a lead in ~/.config/crux/leads.json:
"swarm": { "backend": "exec", "exec": { "command": "tmux new-session -d -s {name} '{command}'" } }
Then: crux swarm up --project acme
Run it twice
Nothing happens twice. You get one project and one config entry, and the second run tells you so:
$ crux discover
✓ acme is a crux project — unchanged.
config ~/.config/crux/leads.json — already had this repo
unchanged means unchanged. A command that reports success while doing nothing is the worst
failure mode there is, so discover distinguishes created, updated (and names the fields it
changed), and unchanged — and if the project already exists it asks before touching it.
The two refusals are both the same mistake caught from opposite sides, and each names its fix:
- the slug is taken by a different repo — the project is fine, the name collides. Pick another:
crux discover ~/src/acme --slug orbit. - the repo is already another project — one directory with two projects is two leads racing one codebase. It tells you which project already owns it.
And a directory that is not a git repo is refused outright, because a lead has to have somewhere to stand.
Get one lead up
One lead, on one project, before you read anything else on this page. Five steps, in order.
You have a project (you ran crux discover) and a repo for it. Now put
a lead on it.
1. Trust the repo. Do this first.
Claude Code asks you to trust a directory it has never seen before, and
--dangerously-skip-permissions does not skip that prompt. A lead started in a repo Claude has
never opened sits at that prompt forever: the process is alive, nothing is listening, and the only
symptom is silence — which is what a healthy parked lead also looks like.
It is the first thing that will bite you, so get in front of it. Once per repo, by hand:
$ cd ~/src/acme
$ claude # accept the folder-trust prompt, then quit
That is the whole fix, and it is permanent. crux does catch the failure afterwards — a lead that
never reaches its tail is reported stalled — but a stalled first
swarm is a bad way to meet the product.
2. Read the plan before you run it
You are about to let a tool spawn long-lived agent processes on your machine, in your repos, with your credentials. Do not take that on faith. Make it show you the plan:
$ crux swarm up --project acme --dry-run
backend: solo · config: ~/.config/crux/leads.json · DRY RUN — nothing will start
START acme would start in /Users/you/src/acme
brief: /Users/you/.config/crux/leads/acme.md
1 would start
--dry-run starts nothing. It prints which project would get a lead, which repo it would run in,
and where its brief would be written. Read that repo path. A wrong one gets you a lead that
reads the wrong code and reports confidently on work it never did, and that is far easier to catch
here than afterwards.
3. Start it
$ crux swarm up --project acme
backend: solo · config: ~/.config/crux/leads.json
START acme started Solo process 41221 in project 3 (acme) — project created
brief: /Users/you/.config/crux/leads/acme.md
1 started
A lead is now running. It read the brief at that path, and it is parked.
4. Confirm it is alive
$ crux swarm status --project acme
backend: solo · config: ~/.config/crux/leads.json
OK acme covered pid 41221 is already tailing "acme". Nothing to do.
1 covered
covered is health. It means something on this machine is really running
crux tail --project acme — not that Solo believes a process is up, but that the OS process table
says so. That distinction is the whole design, and it is worth a section.
If you get stalled instead, you skipped step 1. Go back to it, then run
crux swarm down --project acme and crux swarm up --project acme.
5. Give it something to do
A parked lead makes no noise, so the way to see it work is to hand it work:
$ crux push --project acme "the importer keeps timing out — bump the worker timeout?"
pushed item #41 → acme
Within a poll interval (60s by default) the lead wakes, picks the thread up, and names it:
$ crux feed --project acme
#41 [acme] the importer keeps timing out — bump the worker timeout? 2m ago
The thread is now working, with a title and a status line the lead wrote. Open it on the dashboard
or on your phone and you will watch the lead do one of three things to it: answer it as a chat
message, spawn a worker to write the code, or stop and ask you.
That is a swarm of one. The rest of this page is what it does, and how to run more than one.
What you just started
A swarm is one standing lead per crux project.
A lead is a durable agent, not a cron job and not a queue consumer. It is parked on crux tail, a
long-lived connection that prints one JSON line when something happens on that project and nothing
at all the rest of the time. Silence is the parked state. A parked lead is blocked on a socket: it
is not polling, not burning tokens, and not costing you anything until a real event arrives.
When something does arrive — an item you pushed, a reply you typed on your phone, an answer to a question the lead asked — the lead wakes and does one of three things:
- Answers it, if it can, from what it already knows. The answer lands on the thread as a chat message, and you read it whenever you get round to it.
- Spawns a worker — a separate headless agent that does the code work in your repo, reports back onto the thread, and dies. The lead does not do long work itself, because an agent doing work is an agent that has stopped listening.
- Blocks and reaches you. If the call is yours — scope, money, anything client-facing or
irreversible — the lead raises a typed block with
crux need, and that is the one thing that pushes to your phone.
That is the point of the whole thing. Your agents run without you, and the only thing that gets your attention is the thing that requires it.
The contract every lead follows is not folklore hidden in a prompt. crux swarm up writes it to
disk at ~/.config/crux/leads/<slug>.md, in prose, and starts the agent by telling it to read that
file. It is yours: read it, diff it, edit it, restart the lead.
The generated brief is reproduced in full below.
What "working" looks like
It looks like nothing. This is the part that catches people, so it gets its own section.
A healthy lead with no work to do prints nothing, logs nothing, and costs nothing. It is blocked on a socket. From the outside it is indistinguishable from a lead that crashed at 2am, from a lead frozen at a trust prompt, and from a lead whose tail was killed by its harness twenty minutes ago.
So do not read silence as health, and do not read it as failure. Read crux swarm status.
The thread is quiet, and swarm status says |
It means |
|---|---|
covered |
Working. There was no work. Go away. |
stalled |
Dead on its feet — it never reached its tail. Fix it here. |
tail-down |
Alive, between tail restarts. Normal in passing, a bug if it persists. |
uncovered |
There is no lead at all. Run crux swarm up. |
status exits 1 if any project is uncovered, so it works unattended. Put it in a cron job or a
menubar script:
$ crux swarm status --json | jq -e '.ok'
You want to know your fleet is dead before your fleet's silence convinces you there was no work.
Now do the whole fleet
$ crux swarm up
With no --project, up is a reconciliation, not a spawn. It starts a lead for every project
that does not have one, and it is idempotent: run it twice and the second run starts nothing.
That matters more than it sounds. Two tails on one project race a shared cursor file and deliver every item twice, so "start a lead" has to mean "make sure exactly one lead is running", and it does.
$ crux swarm status
backend: solo · config: ~/.config/crux/leads.json
OK acme covered pid 41221 is already tailing "acme". Nothing to do.
OK orbit covered pid 41244 is already tailing "orbit". Nothing to do.
OK mercury covered pid 41250 is already tailing "mercury". Nothing to do.
OK personal leadless Human-only project. Items are read on the dashboard.
OK unrouted covered pid 41266 is reading the unrouted queue (0 waiting).
Nothing to do.
FAIL quill refused no repo path for "quill" — a lead has to run somewhere.
Add `swarm.projects.quill.repo` to the config, or give it
a project on your backend.
4 covered, 1 leadless, 1 refused
Three healthy projects, one deliberately leadless, one that crux will not guess about — see
a project with no repo is refused — and unrouted,
which is not a project at all. It is the router, and
it is the next section.
Take it down
down stops only the leads crux swarm up started, tracked in ~/.config/crux/swarm.json. A
lead you started by hand is reported and left running:
$ crux swarm down --dry-run
DRY RUN — nothing will be stopped
OK acme not-ours pid 41221 tails "acme" but crux swarm did not start it.
OK orbit not-ours pid 41244 tails "orbit" but crux swarm did not start it.
That restraint is deliberate. A down that swept up every crux tail it could find would kill leads
it knows nothing about, cannot restart, and was never asked to touch. Drop --dry-run to stop the
ones that are crux's.
Both up and down take --project <slug> (repeatable) and --dry-run.
The fence: a new lead retires the old one
Everything above keeps a second lead from starting. This is what happens when one starts anyway —
because you moved the fleet to another machine, because a supervisor came back from the dead, or
because somebody ran crux swarm up on their laptop while the real fleet was running on a box.
The leads crux generates park on a claimed tail:
crux tail --project acme --topics items,activities,decisions --actor lead-acme --claim
--claim posts a claim naming that exact process (host:pid:starttime) before it reads a single
event, and every tail on the stream is delivered it as one more record. Last writer wins: the
lead that started most recently owns the project, and any lead holding an older claim prints the
takeover and exits 75.
So a migration is now: start the lead on the new box. The old one stands down by itself. Nobody has to race anybody, and there is no window where both are working the same board.
Exit 75, and the one thing your supervisor has to know
A supervisor must not restart a lead that exited 75.
75 means another lead owns this stream now. It is not a crash, and a supervisor that treats it as one produces ping-pong: the retired lead comes back, claims, retires the lead that just retired it, that one comes back, and the two of them hand the project back and forth forever — handling every item twice on each pass, while the process table looks busy and healthy. It is worse than the double it was meant to fix.
| Supervisor | What it needs |
|---|---|
| systemd | RestartPreventExitStatus=75 in the unit, alongside Restart=always |
| Solo | no exit-code rule yet — the lead's brief tells it to stop its own process (solo processes stop $SOLO_PROCESS_ID) on the way out |
| anything else | do not restart on 75 |
Two more things worth knowing before you lean on it:
- The retiring lead does not advance its cursor past the takeover. The incoming lead resumes from the same position and replays that batch, so a handover is an ordinary at-least-once replay rather than a hole. On a different machine with no cursor of its own, a claiming lead starts from its own claim — not from "now", which would be decided by its first poll and would pin past anything recorded in between, a rival's claim included. That window was real: it is how the fence's own acceptance test intermittently hung (crux #312), and closing it is the difference between "no takeover was missed" and "no takeover was missed so far".
- A claim is not a credential. Anyone holding your API token can claim your streams; it exists so a lead can tell its own claim from a foreign one, and it authorises nothing.
The router, and the queue nobody was reading
Everything you capture is matched against your projects' keywords by the server. What matches lands on that project's board and wakes its lead.
What does not match lands in one queue. It is called the unrouted queue — crux tail --unrouted
watches it, crux next drains it — and it is where every thought you type that does not obviously
belong anywhere goes to wait for a person.
If nothing is reading it, it waits forever. And you cannot tell, which is the part that matters:
A parked lead is silent. An empty queue is silent. A queue nobody reads is silent in exactly the same way. There is no way to tell "nothing needs you" apart from "everything you captured is stranded" — and the second one looks, from the outside, like a healthy quiet crux.
So crux swarm up starts a router: one standing lead, on the queue instead of on a project.
$ crux swarm up
backend: solo · config: ~/.config/crux/leads.json
START acme started Solo process 41221 in project 3 (acme)
agent: solo agent tool 3 (built-in default)
brief: ~/.config/crux/leads/acme.md
START unrouted started Solo process 41266 in project 4 (unrouted)
agent: solo agent tool 3 (built-in default)
brief: ~/.config/crux/leads/unrouted.md
reads the unrouted queue and routes each item to a
project (or asks you).
~/src/acme is only its working directory — it has no
code and will not touch it.
2 started
What it does, and the one thing it will not do
Its brief is at ~/.config/crux/leads/unrouted.md, and it is short, because the job is:
- Park on
crux tail --unrouted. - Read the item. Read
crux projects— never route from memory; the list changes every time you runcrux discover. - One project clearly fits →
crux route <id> --project <slug> --note "<why>". The project's own lead picks it up from its own tail. The router does not accept it, answer it, work it, or spawn a worker on it — that belongs to the lead that knows the code. - Nothing clearly fits → it asks you, with
crux need, which is the only thing that reaches your phone.
Step 4 is the product, not the fallback:
A mis-routed item is worse than an unrouted one. An unrouted item is findable — it sits in a queue whose entire purpose is to be looked at, and
swarm statuscounts it out loud. A mis-routed one lands on the wrong board, gets accepted and worked by a lead that knows nothing about it, and the person who should have seen it never does.
So the router does not guess. Two plausible projects is not a tie it breaks; it is a question it asks, with both as options, and you answer it in one tap.
It is not a project, and it is not dispatch
The router is never registered as a crux project. A project is a routing target — a router you
can route to is a queue that drains into itself. It lives on the reserved slug unrouted, which is
what the CLI already calls this queue, so the row in swarm status names the thing it covers.
If you already have a lead reading this queue
Say you have a hand-rolled dispatch agent that has been draining the unrouted queue for months.
Tell crux its name, or crux will start a router on top of it and every item in the queue will be
delivered twice:
{ "swarm": { "router": { "slug": "dispatch" } } }
It will not let you find that out the hard way. Coverage is an OS fact —
crux looks at the process table for a crux tail --unrouted — but an unrouted tail is the one tail
that does not name the stream it consumes. crux tail --project acme says which project it is
reading; crux tail --unrouted says nothing, and every crux account's queue tail is byte-for-byte
identical to every other's. So crux matches on the one thing that does identify it, the --actor
its brief makes it announce (lead-<router-slug>), and reports anything else it finds:
WARN unrouted other-router pid 75793 is reading an unrouted queue as `lead-dispatch`, which is not
this crux's router (`lead-unrouted`) — so it is not counted as coverage.
If that is another account on this machine, ignore this. If it is YOUR
queue reader under another name, set `swarm.router.slug: "dispatch"` in
the config, or crux will start a second reader and the queue will be
delivered twice.
crux genuinely cannot tell those two apart — a second crux account on the same machine (which is what
CRUX_CONFIG is for) looks exactly like your own lead under an unfamiliar name. So it does not
guess. It names what it saw and names the one line that settles it.
(This is not theoretical. It is what happened the first time a stranger's swarm was brought up on a laptop that already had a fleet on it: their brand-new crux found the other account's dispatch lead and reported their queue as already double-read. It was being read zero times.)
Once crux knows the name, your existing lead reads as covered, swarm up starts nothing, and
swarm down reports it as not-ours and leaves it running.
Where it runs, and why there
A standing lead is a process, and a process needs a working directory. The router is the one lead with
no repo of its own — it has no code, it reads a queue — so crux discover lends it the first repo
you register, written to swarm.router.repo where you can see it and change it.
That is a deliberate choice. Claude Code will not run in a directory it has never been trusted in, and
--dangerously-skip-permissions does not skip that prompt — so a
directory crux invented for the router would be untrusted by construction, and the router would hang
there on every first run. The repo you just discovered is the one you have already trusted.
It has no work in that repo and its brief says so, in those words: it is not your project, do not read it, do not change it, do not commit to it.
Turning it off, and turning it down
The router reads and routes; it does not write code. A cheaper model is a perfectly good choice:
{ "swarm": { "router": { "agent": "claude --model haiku --dangerously-skip-permissions" } } }
(On Solo, that lever is an agent tool id — "router": { "agentToolId": 5 }. Same reason as
everywhere else.)
To triage by hand instead, say so where every other deliberate absence is said:
{ "leadless": { "unrouted": "I drain it myself with `crux next`." } }
That reads as OK … leadless and crux never mentions it again. What it will not do is let you not
notice.
A scoped run does not go quiet about it
crux swarm up --project acme starts one lead, because that is what you asked for. It does not start a
router — crux does not spawn agents nobody asked for — but it does not shut up about the queue either:
$ crux swarm up --project acme
START acme started Solo process 41221 in project 3 (acme)
WARN unrouted unwatched NOTHING is reading the unrouted queue (2 waiting), and this
run is scoped to "acme" so crux is not starting a router
unasked. Everything captured that matches no project keyword
is stranded there. `crux swarm up` (no --project) starts one.
1 started, 1 unwatched
A warning, not an error — you scoped the run, and it will not fail your exit code for it.
The backends
Bring your own orchestration, but we ship our own opinions.
crux does not want to be your process supervisor. You almost certainly already run agents somehow, so a lead is started through an adapter, and you pick which one.
| Backend | What it is | Needs |
|---|---|---|
solo |
Spawns a Solo agent process per lead, in a pane you can open and read. Nothing to declare first. The only one that can spawn workers — see the note below. | Solo installed |
exec |
Any command template you like — tmux, nohup, systemd, anything with a command line. |
whatever your command names |
You do not have to choose. Set nothing and crux looks at your PATH — Solo, then tmux (as exec,
with a command template it writes for you) — and crux discover records what it found in
your config, so the choice is visible and yours to change. Pin it in config (swarm.backend),
override per run with --backend, or with CRUX_SWARM_BACKEND.
If crux finds none of them, it says so at crux discover — not one command later, after telling
you to start a lead it knew could not start.
Read this before you choose.
crux spawn— how a lead hands real code work to a worker — is Solo-only today. Onexeca lead can still ack a thread, answer it, escalate it, route it, and close it, but it cannot spawn a worker; it has to do the work in its own loop, and the generated brief tells it so in as many words. See Limits.
solo — the default, and the one we run
Solo is a terminal workspace for the processes and agents you already have running. It is where a lot of people keep their agents. It is where we keep ours, and it is the backend crux is opinionated about.
If you are on Solo, this is the entire config:
{ "swarm": { "backend": "solo" } }
Nothing needs declaring first, and no repo paths are needed. Solo already records a path per
project, so an existing Solo user can run crux swarm up having configured nothing at all. And when
the Solo project does not exist yet, the backend creates it from your repo path rather than
refusing to start:
$ crux swarm up
backend: solo · config: ~/.config/crux/leads.json
START acme started Solo process 41221 in project 3 (acme) — project created
brief: /Users/you/.config/crux/leads/acme.md
START orbit started Solo process 41244 in project 4 (orbit)
brief: /Users/you/.config/crux/leads/orbit.md
2 started
Each lead lands in Solo as a named agent process — lead-acme, lead-orbit — that you can open,
read, and talk to. That is worth more than it sounds the first time a lead does something you did not
expect: you go and look at what it is thinking, in the same workspace as everything else you run.
This is the one real advance over the older crux leads --fix, which could only start a lead a
human had already hand-declared in a solo.yml — no use at all to someone building a swarm from
nothing. The solo backend spawns a fresh agent process directly, with no declaration in the way.
Solo is at soloterm.com.
exec — any command
This is what makes "bring your own orchestration" true rather than a slogan. A stranger on plain tmux
gets a working swarm — and gets it without writing any of this, because with tmux on your PATH and
no template set, crux discover derives exactly the pair below and writes it into your config:
{
"swarm": {
"backend": "exec",
"exec": {
"command": "tmux new-session -d -s {name} '{command}'",
"stop": "tmux kill-session -t {name}"
},
"projects": { "acme": { "repo": "~/src/acme" } }
}
}
It is derived, not assumed: crux only writes it if tmux is really installed, because a default
that assumes a binary and then dies is the same dead end one layer down. Anything already in the file
is yours and is never rewritten — screen, systemd-run, nohup, whatever you run.
Note that it is a pair. tmux new-session -d forks: the process crux spawned exits immediately
while the lead lives on inside the tmux server, so crux is left holding a dead pid. stop is what
lets it address the lead by name instead — which is why crux never ships one without the other,
and why swarm down refuses rather than claim a stop it cannot
make.
crux fills in the blanks. The placeholders are:
| Placeholder | Expands to |
|---|---|
{slug} |
the crux project slug — acme |
{name} |
the lead's name — lead-acme |
{repo} |
the repo the lead works in |
{brief_path} |
the generated brief on disk |
{command} |
the whole agent invocation — claude --dangerously-skip-permissions -n lead-acme "Read … and follow it exactly…" |
An unknown placeholder is a hard failure, never an empty string. {brief} instead of
{brief_path} would otherwise start a lead with a truncated command line and no explanation, and a
swarm that silently half-starts is the ambiguity this whole thing exists to remove.
Set stop whenever your command forks — tmux new-session -d, nohup … &. A forking command
means the process crux spawned exits immediately while the lead lives on inside the tmux server, and
crux is left holding a dead pid: a dead handle to a live lead. With a stop template crux addresses
the lead by name and can still kill it. Without one it cannot, and crux swarm down will tell
you it could not, rather than report a lead it did not stop as stopped:
FAIL acme refused the exec process crux started (pid 41221) has exited, but
"acme" is still being tailed — the lead outlived it. That is
what a forking command template (`tmux new-session -d`,
`nohup … &`) does: crux is left holding a dead pid. Set
`swarm.exec.stop` (e.g. "tmux kill-session -t {name}") so
crux can name the lead instead of the process, then run
`crux swarm down` again. Nothing was stopped.
Writing your own
A backend is three methods and one declaration. That is the entire contract:
{
name: 'mything',
// Can this backend name a lead forever, or only by pid?
// 'handle' — the ref alone addresses the lead for good (a Solo process id, a tmux
// session name). The recorded process dying does not mean the lead died.
// 'pid' — the ref IS an OS pid, and if it dies there is no other way to name the
// lead. `swarm down` then REFUSES rather than pretend it stopped something.
addresses: 'handle',
start(lead) { // start one standing lead
// lead: { slug, repo, prompt, briefPath, agent, … }
return { ref: 'opaque-handle', detail: 'human-readable, shown in `swarm up`' };
},
stop(entry) { // → 'stopped' | 'already-gone' | 'unknown'
},
list() { // what this backend can see right now
return [{ slug: 'acme', ref: 'opaque-handle', running: true }];
},
}
Note what is deliberately not in the interface: coverage. A backend is never asked "is acme
covered?" — see below. It is the idea the rest rests on.
Config
Config lives in ~/.config/crux/leads.json, or the cli/leads.config.json shipped beside the CLI,
or wherever CRUX_LEADS_CONFIG points. First one found wins.
You do not have to write any of it by hand. crux discover writes the
one key a lead cannot start without — swarm.projects.<slug>.repo — and creates the file if you have
none, inheriting whatever was in force so nothing you already had stops applying. Read this section
when you want to change something discover does not ask about.
On the solo backend, the whole file can be this:
{ "swarm": { "backend": "solo" } }
Everything else is there when you need it:
{
"map": {
"acme-search-index": "acme",
"status-page": "beacon"
},
"leadless": {
"personal": "Human-only project. Items are read on the dashboard, not routed to an agent lead."
},
"swarm": {
"backend": "solo",
"agent": "claude --dangerously-skip-permissions",
"solo": { "agentToolId": 3 },
"exec": { "command": "…", "stop": "…" },
"projects": {
"acme": {
"repo": "~/src/acme",
"brief": "Extra context pasted into this lead's generated brief.",
"agentToolId": 7
},
"orbit": {
"repo": "~/src/orbit"
}
},
"router": {
"repo": "~/src/acme"
}
}
}
map — your backend's project name → the crux slug, where they differ. This is explicit config,
not string munging: no transformation derives acme from acme-search-index. If the names are
already equal, you do not need an entry.
leadless — projects that should not get a lead, and why. The reason is required, because
it is what swarm status prints, and a project quietly absent from the reconciliation is
indistinguishable from one that is broken. "unrouted" here turns
the router off.
swarm.projects.<slug>.repo — where that lead runs. Required for any project your backend cannot
locate itself. Solo already records a path per project, so a Solo user often needs none of this;
exec users need one per project.
swarm.router — the router, which is a lead but
not a project. repo is a working directory (crux discover writes your first one here); agent /
agentToolId choose its model, exactly as for a project; brief adds context to its generated brief;
slug moves it off the reserved word unrouted, which you need only if you have a crux project by
that name — and crux refuses loudly rather than let the two quietly take each other's lead.
A different agent (or model) per project
Not every project deserves the same model. One is under active development and wants the strongest agent you have; three are parked on a tail, waking twice a week to ack a thread and route it. Running those three at the same rate as the first is just money.
So the agent is a per-project override, falling back to a global, falling back to the default:
swarm.projects.<slug>.<knob> → the global → the built-in default
There are two knobs, and which one is real depends on your backend. This is not a style choice — the backends differ in kind:
| Backend | The lever | Why |
|---|---|---|
exec, herdr |
agent — a command |
crux builds the command line and they run it, so a model is just a flag on it. |
solo |
agentToolId — a number |
Solo owns the command. solo processes spawn takes an --agent-tool-id and looks the command up in its own registry of agent tools; there is no flag that takes a command line. So on Solo you choose the tool, not the command. |
On exec or herdr:
"projects": {
"acme": { "repo": "~/src/acme", "agent": "claude --model opus --dangerously-skip-permissions" },
"orbit": { "repo": "~/src/orbit", "agent": "claude --model haiku --dangerously-skip-permissions" },
"mercury": { "repo": "~/src/mercury" }
}
mercury gets swarm.agent if you set one, and claude --dangerously-skip-permissions if you did
not.
On solo, the same idea with the lever Solo actually has. solo agents list prints the ids:
$ solo agents list
1 Gemini gemini
3 Claude claude --dangerously-skip-permissions
4 Codex codex
"swarm": {
"backend": "solo",
"solo": { "agentToolId": 3 },
"projects": {
"acme": { "agentToolId": 7 },
"orbit": { }
}
}
acme runs Solo's tool 7; orbit and everything else run tool 3. To put a project on a different
model here, add the model to a tool in Solo and use its id.
A knob your backend cannot read is REFUSED, not ignored
Set agent on a project while running solo and crux will not start that lead:
FAIL acme refused `swarm.projects.acme.agent` is set to "claude --model opus",
but this swarm runs on the solo backend, which does not read
`agent` — NOTHING is applying that line. A lead started from
this config runs the default agent and looks perfectly healthy.
On solo, use `swarm.projects.acme.agentToolId: <id>` — Solo owns
the command, so on solo you pick the AGENT TOOL, not the command
line. `solo agents list` prints the ids (3 is `claude
--dangerously-skip-permissions`).
This is the whole point of the feature, not a rough edge of it. Accepted quietly, that lead would come
up on the default agent, report started, sit there in swarm status as covered, and bill at the
rate you were trying to escape — while the config line you wrote to change it did nothing at all. A
no-op that looks like a success is the failure this codebase keeps finding, and the refusal is what
closes it. The same goes the other way: agentToolId on an exec lead is refused too.
It is refused even when the lead is already up and healthy. Most refusals are checked after
coverage — there is nothing to refuse about a project crux was not going to touch. This one is checked
before it, deliberately: a broken config line is broken whether or not a lead happens to be running
on it, and a conflict that only spoke up for uncovered projects would fall silent the moment the swarm
came up — leaving covered, exit 0, and a config line doing nothing, invisibly, forever. Fix the
config; the refusal goes away.
A global swarm.agent on solo is a warning rather than a refusal — the global block legitimately
configures every backend at once — but it is not silent:
WARN `swarm.agent` is set to "claude --model opus", and the solo backend does not read it — Solo owns
the command … To choose the agent on solo, set `swarm.solo.agentToolId` (all leads) or
`swarm.projects.<slug>.agentToolId` (one lead) — `solo agents list` has the ids.
A config change does not touch a lead that is already running
It takes effect on the next swarm up. That is correct — rewriting a live lead's command out from
under it would be far worse — and it is surprising, because you edit the config, run swarm status,
and see the value you just typed. So when the two disagree, status says which is which:
OK acme covered pid 4242 is already tailing "acme". Nothing to do.
agent: solo agent tool 9 (swarm.projects.acme.agentToolId)
the RUNNING lead was started with solo agent tool 3 — a
config change does not touch a lead that is already up.
It takes effect on the next `crux swarm up`
(`crux swarm down --project acme` first, if you want it now).
A project with no repo is refused, not skipped
FAIL quill refused no repo path for "quill" — a lead has to run somewhere.
Add `swarm.projects.quill.repo` to the config, or give it
a project on your backend.
A lead has to run somewhere. There is no sensible default, and guessing at one gets you a lead reading the wrong code. So a project crux cannot place is named, explained, and counted in the exit code. It never falls silently out of the plan.
Usually this means the project was created by the router — which knows what a project is but has no
idea where its code lives on your machine. The fix is to tell it, and that is crux discover:
$ crux discover ~/src/quill --slug quill
The same goes for a repo conflict: if your backend already runs that project's agents in one directory and your config claims another, crux refuses rather than start a lead whose brief describes a repo it is not standing in. Every symptom of that points somewhere other than the config line that caused it.
Reading swarm status
| State | Means |
|---|---|
covered |
Something is tailing this project. This is health. |
leadless |
Deliberately has no lead, per config. |
starting |
crux started this lead and its process is alive, but its tail has not appeared yet. Normal for a few seconds. |
tail-down |
The lead is alive and has tailed before, but nothing is tailing right now — most likely between tail restarts. Warned about, not acted on. |
not-ours |
Something is tailing this project, but crux did not start it. Left running. |
uncovered |
Nothing is tailing it. swarm up would start a lead. Exits 1. |
unwatched |
Only the router: the unrouted queue has no reader, and this --project run is not going to start one. A warning; exits 0. |
other-router |
Something is reading an unrouted queue under a name that is not this crux's router. Another account — or your own lead. Never counted as coverage, in either direction. A warning; exits 0. |
stalled |
Alive, but it has NEVER tailed. A dead swarm that looks alive. See below. |
refused |
crux will not act, and says why. |
The unrouted row is the router, and it carries one
extra fact nothing else does — how deep the queue is:
OK unrouted covered pid 41266 is reading the unrouted queue (3 waiting).
FAIL unrouted uncovered NOTHING is reading the unrouted queue (3 waiting). Everything
captured that matches no project keyword is stranded there,
silently. `crux swarm up` would start a router.
That number is the whole point of the row. "Nothing is reading your queue" is a claim about processes, and a user with an empty queue would rightly shrug at it. "Three of the things you captured are sitting in it unread" is a claim about them.
crux swarm status --json carries it too, as queue: {slug, state, depth} — and it exits non-zero
on an uncovered lead, exactly as the human output does. (It did not, once. A monitor is precisely the
thing that reads the JSON, and it was reporting a dead fleet green.)
Under every project that has a lead, status prints which agent it will run, and the config line that decided it:
OK acme covered pid 4242 is already tailing "acme". Nothing to do.
agent: solo agent tool 7 (swarm.projects.acme.agentToolId)
OK orbit covered pid 4310 is already tailing "orbit". Nothing to do.
agent: solo agent tool 3 (built-in default)
That second line is not decoration. A default that lives only in crux's head is one you cannot see,
cannot check, and cannot tell apart from a choice you made yourself — so crux says built-in default
out loud, exactly as discover does for the backend it works out for you. Setting it is
here.
Coverage is an OS fact
Here is the design idea that makes everything above work, and it is worth understanding, because it is what you are trusting.
crux never asks the backend whether a lead is healthy. It reads the OS process table and asks a single, dumb, checkable question:
Is something on this machine actually running
crux tail --project <slug>?
That is coverage. Not "does Solo say the process is running", not "does the multiplexer list an agent". Those answers cannot be trusted, and not hypothetically:
- Solo's own status can lie. A process it lists as running can be gone. We like Solo. We still do not ask it this question.
- tmux has never heard of a crux slug. It has no opinion to give.
- An agent multiplexer reports a healthy parked lead as
idle, because parking is doing nothing — and it reports a lead frozen at a trust prompt asidletoo. The two states that matter most, the lead working perfectly and the lead dead on its feet, are indistinguishable to it.
The process table separates them cleanly, because one of them is running crux tail --project <slug>
and the other never got there. So coverage comes from the process table for every backend, and
the adapter is never asked.
Two things fall out of this, and both are load-bearing:
- The
execbackend gets working reconciliation for free. crux does not need to understand tmux to see the tail that tmux started. - Every failure below becomes visible, instead of being papered over by a backend cheerfully reporting green.
When it goes wrong
Two failures. Both of them look like silence. Both were found by running a real swarm, not by reasoning about one.
The lead that never tailed
stalled — the one that will bite you first, and step 1 is
how you avoid it.
Claude Code asks you to trust a directory it has never seen before, and
--dangerously-skip-permissions does not skip that prompt. So a lead started in an untrusted
repo sits there forever: the agent process is alive, the swarm is dead, and the only symptom is
silence. swarm up will have told you "1 started" and been technically right.
crux catches it. A lead that is alive but has never reached its tail, past a 3-minute grace
window, is reported stalled:
FAIL acme refused its lead (812) has been up for 7m and has NEVER started its
tail — it is stuck before `crux tail --project acme`, so
nothing is consuming the stream. The usual cause is the agent
sitting at a prompt: Claude Code asks you to trust a directory
it has not seen before, and `--dangerously-skip-permissions`
does NOT skip that one. Read its output, or run `claude` once
in the repo and accept the folder. Not starting a second lead
(it would stall too) — `crux swarm down --project acme` clears it.
To clear it: run claude once, by hand, in the repo, and accept the folder. Then:
$ crux swarm down --project acme
$ crux swarm up --project acme
The lead that stopped tailing
tail-down. An agent harness kills a long-running foreground command when its timeout expires.
Claude Code's Bash tool does this after about two minutes. Your crux tail is a long-running
foreground command.
So a lead that runs crux tail and treats its return as the end of the job goes deaf without
dying. The process is up, the agent is idle, nothing is consuming the stream, and the symptom is
silence again.
The generated brief tells the lead this in bold, twice, and gives it both remedies:
- Arm a persistent monitor on the tail (in Claude Code: the
Monitortool) and let it wake you. This is the cheap, correct way to park. - Or run the tail with the longest timeout the harness allows and re-run it immediately every time it exits. Forever.
If you write your own brief, keep this. It is not optional. A lead whose tail is not running is not a lead.
crux surfaces the aftermath as tail-down rather than starting a second lead on top of it — the live
one owns that slug. Seen once, in passing, it is a lead restarting its tail. If it persists, read
the lead's output: it has stopped restarting.
The lead that exited 75
It was superseded — another lead claimed the project, and this one stood down on purpose. Its last line says so:
crux tail: superseded — lead-acme claimed "acme" as box:4412:1753900000000. Standing down (exit 75);
this tail must NOT be restarted.
This is the fence working, not a failure. The project is covered by whoever claimed last; check with
crux swarm status, and if that says uncovered, the claimer died right after claiming — start a
lead and it will win the stream on its way up.
The one thing to look for is a lead that exits 75 over and over. That is ping-pong: something is restarting a superseded lead, so the two of them are trading the project back and forth and handling every item twice on the way. Fix the supervisor, not the lead — exit 75 and what your supervisor has to know.
The brief is the contract
crux swarm up generates this and writes it to ~/.config/crux/leads/<slug>.md before starting the
agent. The agent is started with a one-line prompt that says, essentially, read that file and follow
it exactly.
Every line in it is there because getting it wrong broke something real. It is reproduced in full so you know what you are running — and so you can decide to change it.
# Standing lead: acme
You are the durable PROJECT LEAD for **Acme** (crux slug `acme`), working in `/Users/you/src/acme`.
You run forever.
## Your loop (never exit)
1. Park on the tail. It prints one JSON line per event you should wake on, and NOTHING
otherwise — silence is your zero-token parked state, not a reason to poll:
crux tail --project acme --topics items,activities,decisions --interval 60 \
--actor lead-acme --claim
All three topics are required. A human reply arrives as an **activity**, and the answer to a
`crux need` block arrives as a **decision**. On `items` alone you would hear the question and
never the answer, and the thread is one-way.
`--actor lead-acme` registers your live presence, so the dashboard can tell "no items" apart
from "the lead is dead" — the failure that once went unnoticed for hours.
`--claim` fences the stream: it retires any other lead on this project the moment you start,
and it is how you find out if one retires you. Read the takeover section below before you
restart a tail that exited 75 — that exit code is the one time "start it again" is wrong.
2. **PARK DURABLY — read this twice.** Your harness will kill a long-running foreground command
when its timeout expires. That is normal, it is not an error, and it is **not the end of your
job**: the moment the tail returns, START IT AGAIN. Forever.
[ … the two remedies — Monitor, or a restart loop … ]
A lead whose tail is not running is not a lead. Nothing is consuming the stream, and from the
outside that is indistinguishable from "there is no work" — which is exactly how a dead fleet
goes unnoticed for hours. Your tail being up IS the job.
3. When a line arrives, handle it (below), then go back to waiting. **Never exit.**
Delivery is at-least-once — the cursor is durable, so a restart replays rather than drops. An
item can therefore repeat. Before acting on an item id, check whether you already logged a
`crux activity` for it.
## Handling an item
- **Ack on receipt, and NAME IT.** The instant a thread arrives:
crux accept <id> --actor lead-acme --title "<2-3 words>" --status "<what you are doing>"
Accept flips the thread out of idle into "working" — do it BEFORE the work, not after, or the
thread reads as untouched while you are busy on it.
`--title` is the thread's NAME and it is the row's main line from now on. **Nothing generates
it but you.** There is no model on the server deriving titles from the captured text — you have
just read the thread and you are about to work it, so you already know what it is called, and a
thread you accept without `--title` stays untitled forever (its row goes on showing the raw
captured text, which is what every unaccepted thread shows). Two or three words: "auth timeout",
"invoice rounding". It is a name, not a summary — the row clips it to one line.
- **Answerable from what you know** → reply as chat:
crux activity --item <id> --project acme --actor lead-acme --type note "<your answer>" \
--status "<what is happening now>"
Always pass `--item <id>`, or the reply is filed against the project and the item view never
shows it. `--type note` is what renders as a chat bubble.
- **Real code or task work** → spawn a worker; do not do long work inside your own loop, or you
stop listening while you do it.
crux spawn <id> --project-id 3 --brief "<the self-contained task — crux gives it the worktree>"
The worker runs HEADLESS: it works the thread, exits, and crux posts its final message back
onto this thread automatically — plus a failure note and a push if it crashes or says nothing.
Its report wakes you. Relay it, or act on it.
**`crux spawn` makes the worker a WORKTREE and starts it inside one.** You do not ask it to,
and you must never tell it to make one itself: a brief that says "work in a git worktree" is a
brief a worker can satisfy by running `git checkout -b` in YOUR checkout, which leaves you
committing to its branch and `git push origin main` exiting 0 having pushed nothing. That has
happened twice. Brief the WORK; the worktree is the harness's job now.
- **The human's call** (scope, money, client-facing, irreversible) → raise a typed block:
crux need --item <id> --kind decide|answer|secret|desk|world "<one-line question>" \
--actor lead-acme --status "<why it stopped>"
A typed block is the ONLY thing that reaches their phone. A plain activity does not, and a raw
"NEEDS YOU:" note is not an escalation — the escalations view cannot render it.
- **Misrouted** → `crux route <id> --project <other-slug> --note "<why>"`.
## The row is yours to write
Every thread shows up as a ROW: a short **title** on the main line, and under it, smaller and in
grey, a **status line** — one phrase saying what is happening to it right now.
**You write both. Nothing else does.** The server does not generate them; there is no model on the
write path re-reading the captured text to guess a title. There does not need to be one — you are
already an LLM, you have already read the thread, and you already know what you are doing to it.
Deriving that a second time, from less, would cost more and know less.
So the rule is: **you never make a separate call to update the row.** You carry `--status` on the
request you were already sending. It is one flag on the verbs you already run:
```
crux accept <id> --title "<name>" --status "<phrase>" # name it as you pick it up
crux activity --item <id> --status "<phrase>" "<text>" # ...as you report progress
crux spawn <id> --title "<name>" --status "worker on it"
crux need --item <id> --status "blocked on the DNS answer" ...
crux done <id> --status "merged, closed" # a RESTING phrase — see below
```
That is why a status line cannot go stale: you set it AS you work, so it always says what you last
said it says. A status line the server derived would be wrong the moment you did the next thing.
Keep it short and keep it TRUE-NOW ("running the migration", not "will fix the migration"). On
`crux done` the status line should describe REST, not work — "merged, closed", "shipped", "not
reproducible". Omit it there and crux writes "done" (or "dropped") itself; either way a closed
thread never sits there still announcing work that stopped hours ago.
## How you write
Everything you post is read in a chat bubble, on a phone. Four rules, and they cost nothing:
1. **Never name a PR, issue, or thread without its link.** A bare `#29` is something the reader
has to go and find; a link is something they tap. `gh pr view <n> --json url -q .url` gets it,
and `crux activity` renders markdown — `[#29](<url>)` works.
2. **Use real formatting** — bold the thing that matters, a table to compare two options, a code
fence for commands and their output. A wall of prose is a wall.
3. **Answer first, detail after.** A phone shows a few lines before the fold, and the first ones
have to be the ones worth reading.
4. **A thread is `#41`; a TASK is `!41`.** Two tables, two id sequences, both starting at 1 — so
"task #41" is a reference that LINKS to thread 41, and a wrong link is worse than no link
because a wrong one gets followed. Write `!41` and every surface renders it as a link to the
task, when it is a task you may open.
This is not only about your own notes: **every brief you write for a worker inherits it**, because
a worker's final message is posted to the thread verbatim. Say it in the brief.
## A closed thread refuses to be worked
`crux done` is a real ending. A `done` or `dropped` thread will not let you `accept` it, `spawn`
onto it, or raise a `need` on it — all three exit non-zero and tell you so. This is not pedantry:
accepting a closed thread used to report success and change nothing anyone could see, so the board
went on saying **finished** while a worker built on it and reported into a thread nobody reads.
Follow-up work on a thread you already closed is one of two things, and you say which:
```
crux push --project <slug> "…follow-up, from #<id>" # usually this: new work, new thread
crux accept <id> --reopen --title "…" --status "…" # or bring the old one back, on purpose
crux spawn <id> --reopen --project-id <n> --brief … # same flag, if a worker goes straight on
```
`--reopen` flips the thread back to `routed`, un-archives it, drops the resting status line, and
writes a **reopened** line onto the thread — so it reads `done` → `reopened` → `accepted`, and
never "finished, with a worker attached".
## An agent asked for from the web
A person can ask for an agent on a thread from the browser or the phone. **The request does not
start anything** — the server has no route to the box Solo runs on. **You are that box.** So the
request is a line on your tail, and you are the only thing in the system that can honour it:
```json
{"topic":"activity.created","id":2540,"item":230,"type":"agent.requested","actor":"you", …}
```
`"type":"agent.requested"` is the whole signal — gate on it, and use `item` for the thread id
(the `id` on an activity line is the ACTIVITY, and no endpoint takes one of those). When one
arrives:
1. **Check whether the thread already has a worker** before you spawn — delivery is at-least-once,
so this line can repeat. `crux spawn` refuses a second worker on a thread that has a live one
and tells you so, which is the defence that actually holds; reading first is how you avoid
needing it.
2. **Spawn it**, with a real brief — the request is a person pointing at a thread, not a brief.
Read the thread, write the brief, then spawn as you would for work you picked yourself.
3. **If you are NOT going to spawn** — the thread is already answered, it is a question you can
answer in a sentence, it needs a decision first — **say so on the thread**. The web shows
"requested" until an agent exists, so a request you silently declined reads as a lead that
never woke up:
crux activity --item <id> --actor lead-acme --type note "<why no worker>" --status "<what instead>"
The browser deliberately does **not** show "started" when the button is pressed — only when
`agent_process_id` is set, which is something only your `crux spawn` writes. Until you act, the
thread honestly says nobody has. And a request written while you were down is **not lost**: your
cursor is durable, so it replays to you when you come back up. Work the backlog, do not assume it
was handled while you were away.
## A close you did not make
`crux done` closes a thread and stops its agent in one move: a thread's death is its worker's.
A close made from the WEB or the phone can only do the first half — the server has no way to
reach the box the agent is running on. **That box is yours.** So when an item line arrives on your
tail reading `done` or `dropped` and it was not you that closed it, finish the job:
```
crux bury <id> --actor lead-acme
```
It stops the process the thread recorded, takes the worker's desk away, and removes its worktree
if that is clean and merged. Safe to run twice: stopping a process that has already exited is a
no-op and writes nothing to the thread, so on a headless worker — already dead, that is how its
report got posted — it will usually just tell you so. The case it exists for is a `--live` worker.
That one never exits on its own and has no supervisor watching it at all, so without this it sits
there idling against a thread the board says is finished, and nobody finds out.
It REFUSES an OPEN thread, so a close that was reopened before you got to it cannot take out live
work. And if it cannot reach Solo it says so **on the thread** and exits non-zero — pass that on,
because an agent outliving its thread quietly is the entire failure this prevents.
## If another lead takes this project (exit 75)
`--claim` on your tail fences the stream: it retires any other lead reading it, and it means any
lead that starts after you retires **you**. Last writer wins. That is how a lead moves between
machines without a human racing to kill the old one, and how an accidental second start stops
being a silent DOUBLE where every item is handled twice.
When it happens your tail prints one line and exits **75**:
```json
{"topic":"lead.claimed","instance":"box:4412:1753900000000","project":"…","actor":"lead-…"}
```
Exit 75 means **another lead owns this stream now**. It is not a crash and not an error.
1. **Do NOT restart the tail.** Everything above tells you to restart it forever; this is the one
exception, and it is absolute. Restarting means you claim, the other lead exits, it restarts,
it claims, you exit — ping-pong, forever, with every item handled twice on the way through.
2. **Leave the work.** Do not finish the item you were on. The tail deliberately did not advance
its cursor past the takeover, so the incoming lead replays it. It is covered; you are not.
[ … rules 3 and 4 — post one note, then end your run with status 75 … ]
**Your supervisor must not restart you on 75** (systemd: `RestartPreventExitStatus=75`; Solo:
stop your own process as above).
## Hard rules
- **Finish means CLOSED.** The moment a thread is handled:
`crux done <id> --note "<why>" --status "<resting phrase>"` (`--drop` for won't-do). An unclosed
thread reads as still-in-progress forever.
- **Do not work a closed thread.** Push a new item, or `accept --reopen` and mean it. See above.
- **A close you did not make still needs reaping:** `crux bury <id>`. The web can close a thread
but cannot stop its agent, and you are the only one who can. See above.
- **An `agent.requested` line is a person asking you to spawn.** The web can ask for an agent and
cannot start one; you are the only thing that can. Spawn it, or say on the thread why not. See
above.
- **Stop a worker with `crux cancel <id> --note "<why>"`, never a bare kill.** A killed worker is
indistinguishable from a crashed one — the supervisor sees a process that ended having printed
nothing, and posts "Worker failed" to the thread with a push. `crux cancel` leaves a tombstone
first, so the death is reported as deliberate. A crash note that cries wolf gets the real ones
ignored.
- **A blocked worker must block LOUDLY.** Headless means nobody is watching its prompt. If it is
stuck it must `crux need`; waiting silently is death, not patience.
- **Link everything you name.** A PR, issue or thread without its URL is homework for whoever is
reading. Bold what matters, fence the commands, write it phone-sized. See above.
- **Never post to external services** (Slack, email, GitHub comments). `crux activity` is your
reporting channel. Client-facing asks are always human-in-the-loop: escalate, never answer.
- **Stay inside `/Users/you/src/acme`** (worktrees included). Never commit secrets.
- **Push only the branch you are STANDING on.** `git push origin main` pushes the local `main` ref,
not `HEAD` — from a checkout sitting on another branch it exits 0 and sends nothing, and your
work is stranded on a machine nobody backs up. crux installs a `pre-push` hook that refuses it
(`crux guard`), but the habit is the real fix: `git push origin $(git branch --show-current)`.
- **Never exit** — with exactly one exception: a foreign takeover (exit 75), above. Every other
return is a lead that stopped listening, silently.
---
Generated by `crux swarm up`. Edit this file and restart the lead to change the contract.
Add your own project context to it with swarm.projects.<slug>.brief in the config, which is pasted
into the generated brief. Or edit the file and restart the lead — it is yours. swarm up never
rewrites the brief of a project that already has a lead running.
What happens to a worker
When a lead spawns a worker, that worker's run ends in one of these ways, and the supervisor — not the agent — decides which and posts it to the thread. The agent cannot forget to report, because its final message is the report.
| Ending | What happened | The thread gets | Pushes to your phone? |
|---|---|---|---|
| Reported | The worker printed a result | its final message, verbatim | no |
| Died | The process ended with no usable result | Worker failed — … | yes |
| Cancelled | The same, but the lead claimed it with crux cancel |
Worker cancelled by … Not a crash. | no |
| Report unreadable | A result WAS printed and could not be read whole | Worker finished, but its report could not be read back… Not a crash. | no |
Silence is the bug. A process that ends having printed nothing has told the thread nothing, and
the thread would wait on it forever — so the supervisor speaks for it. This is why a lead must stop a
worker with crux cancel and never a bare kill: a killed worker is indistinguishable from a crashed
one, and a crash note that cries wolf gets the real ones ignored.
But "crux could not read it" is not "the worker printed nothing." Those two shared one message
until three finished workers were announced as crashes in one night, and the difference is now
checked directly: an unescaped "type":"result" in the buffer is proof the run got as far as
printing a result, whatever else is missing. When it is there, the thread is told the report is
partial and shown what survived — never that the worker died, and never with a push.
The full rules are in briefs/WORKER-CONTRACT.md.
The worker's worktree, and the guard on the lead's checkout
Twice in one night, a spawned worker ran git checkout -b in the lead's own checkout and left it
standing on the worker's branch. Everything the lead committed afterwards went there. And then:
$ git push -q origin main
$ # exit 0. no output. nothing wrong on the surface.
git push origin main pushes the local main ref, not HEAD. With HEAD on someone else's
branch, that command pushes a main that has not moved: a no-op that is indistinguishable from a
success. It stranded four commits the first time and the Maestro gate fix the second. Both times the
work was recoverable and prod was never wrong. Both times it was caught by luck.
The old brief told workers to make their own worktree. Telling is what failed. There are now two independent mechanisms, because either one alone would have failed on one of the two nights.
1. crux spawn hands the worker a worktree. Before the agent exists, spawn creates
<repo>-worktrees/thread-<id> on branch thread-<id> (--branch renames it; the path always comes
from the thread id, so cleanup can find it later). Solo runs a process in its project's directory and
there is no --cwd on solo processes spawn, so spawn also points a throwaway Solo project — crux worker #<id> — at the worktree and starts the agent in that. The worktree is not somewhere the
worker is asked to go. It is the only place it has ever stood.
- Respawn (a worker died; a lead re-briefed) reuses the same worktree. It never makes a second.
- A dirty lead checkout is refused, naming the files — a worktree branches from
HEAD, so the worker would silently not have that work.--allow-dirtyif you mean it. crux leadsignores worker desks. They are a cwd, not a repo anybody leads; counted as leads, every spawn would have manufactured anunmappedentry and turned the fleet check red.
Cleanup: crux done does it, and only when it is safe. The desk always goes — the worker is
dead and a Solo project with no process is clutter. The worktree goes only if it is clean and its
branch is merged, because a worker's branch is usually merged by the lead later, and deleting on
close would delete unmerged work as a matter of routine. An unmerged worktree is kept, loudly, with the
one command that removes it. (That is how seventy of them accumulated. It is also why deleting them
blindly would have been worse.)
2. The lead cannot push a ref it is not standing on. crux guard installs a pre-push hook in the
repo's common git dir — so it covers the main checkout and every worktree — and crux spawn installs
it in the repo it spawns for. It refuses two things:
- pushing a branch that is not the one
HEADis on, naming both; and - a push that sent nothing while the checkout is holding commits that are on no remote. This is the
one that catches the actual incident: a stale
mainhas nothing to send, so git says "Everything up-to-date", exits 0 — and calls the hook with empty stdin. There is no ref to object to. The only evidence is that you just "pushed" while your work sits on a branch you are not looking at.
CRUX_ALLOW_CROSS_BRANCH_PUSH=1 overrides it, deliberately, in the transcript.
Limits
crux spawn is Solo-only today. This is the one to know before you pick a backend.
A lead's ability to hand real code work to a headless worker goes through crux spawn, and crux spawn needs a Solo project id. It has no backend abstraction yet. So:
solo |
exec |
|
|---|---|---|
| Start a standing lead | yes | yes |
| Ack, answer, route, close a thread | yes | yes |
Escalate with crux need |
yes | yes |
| Spawn a worker for code work | yes | no |
An exec-backed lead is not broken. It will ack your thread, answer what it can, block when it
should, and close what it finishes. But it cannot delegate: it has to do the work inside its own
loop, and while it is working it is not listening.
crux does not hide this from the lead either. When the swarm is not on Solo, the generated brief says so to the agent's face:
NOTE: `crux spawn` needs a Solo project id and this swarm is not running on Solo, so worker
spawning is unavailable to you. Do the work yourself in the repo above, and keep it short —
every minute you spend working is a minute you are not listening.
If you want the full lead-plus-worker model today, run the solo backend. If you want a swarm on the
orchestrator you already use, exec gives you standing leads that answer and escalate, and worker
spawning will follow.
Reference
crux discover [path] [--yes] [--json]
[--slug <s>] [--name <n>] [--color <c>] [--keywords <a,b>] [--description <d>]
crux swarm up [--backend solo|exec] [--project <slug>]… [--dry-run] [--json]
crux swarm status [--backend solo|exec] [--project <slug>]… [--json]
crux swarm down [--project <slug>]… [--dry-run] [--json]
discover flag |
|
|---|---|
--yes |
Take every default and ask nothing. |
--json |
Machine output. Implies --yes. |
--slug <s> |
Override the derived slug — also how you point at a project you already have. |
--name <n> |
Override the derived display name. |
--color <c> |
emerald | violet | sky | amber | rose | pink | zinc. |
--keywords <a,b> |
Comma-separated, replacing the derived list. |
--description <d> |
One line. The router renders it into its prompt, so keep it a summary. |
| Flag | |
|---|---|
--backend <b> |
solo | exec. Default: config swarm.backend, or CRUX_SWARM_BACKEND. |
--project <slug> |
Only this project. Repeatable. Default: every project. |
--dry-run |
Print the plan and start (or stop) nothing. |
--json |
Machine-readable report. |
status exits 1 if any project is uncovered — or if nothing is reading the
unrouted queue. --json exits the same way, and says
ok: false. up exits 1 if anything was refused — one project failing to start never aborts the
others, but it is always loud, because a half-started swarm that reports success is the worst outcome
available.
| Config key | |
|---|---|
swarm.backend |
solo | exec | herdr. Absent means "look at my machine". |
swarm.agent |
The lead command, fleet-wide. Read by exec and herdr. Not by solo — see below. |
swarm.solo.agentToolId |
The Solo agent tool, fleet-wide. Default 3. solo agents list has the ids. |
swarm.exec.command / .stop |
The exec templates. Set stop whenever your command forks. |
swarm.herdr.session |
Put the leads in a named herdr session, isolated from your own. |
swarm.router.repo |
Where the router runs. crux discover writes your first repo here. |
swarm.router.agent / .agentToolId |
Its model. It reads and routes; it does not write code, so a cheaper one is a fair choice. |
swarm.router.slug |
Move it off the reserved unrouted, if you have a crux project by that name. |
leadless.unrouted |
Turn the router off, and say why. |
swarm.projects.<slug>.repo |
Where that lead runs. Required unless your backend knows. |
swarm.projects.<slug>.brief |
Extra context pasted into that lead's generated brief. |
swarm.projects.<slug>.agent |
That lead's command — exec and herdr only. Per-project agents. |
swarm.projects.<slug>.agentToolId |
That lead's Solo agent tool — solo only. |
Each agent knob overrides its global, which overrides the built-in default. Set the one your backend
cannot read — agent on solo, agentToolId on exec — and that project is refused, by name,
with the knob that works. It is never accepted and ignored.
| Path | |
|---|---|
~/.config/crux/config.json |
the server and your API token, written by crux install ($CRUX_CONFIG). 0600 |
~/.config/crux/leads.json |
config (or cli/leads.config.json, or $CRUX_LEADS_CONFIG) |
~/.config/crux/leads/<slug>.md |
the generated brief, one per lead ($CRUX_SWARM_BRIEFS) |
~/.config/crux/swarm.json |
what crux started — how down knows what is its to stop ($CRUX_SWARM_STATE) |
What to read next
cli/README.md— every verb an agent drives crux with:push,tail,accept,spawn,need,done,cancel.briefs/WORKER-CONTRACT.md— how a spawned worker's run is allowed to end, and why finishing quietly is not one of the ways.