Design: true recovery for sync /v1/runs leaves after a worker/supervisor crash
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 35/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- redis, typescript
- Domain
- backend, distributed-systems
Research direction
Start with the run lease and reaper in harness/src/run-lease.ts, then read the detachable-turn design at docs/specs/2026-10-09-turn-reattach-design.md and the spawn flow in packages/mocactl/src/subagent.ts. Compare the recovery options against credential handling, sandbox side effects, and fencing against a slow original worker. Done means a design decision covers recovery behavior, result delivery, credentials, and the #558 crash reproduction.
Written by the indexing model from the issue text.
Description
Follow-up to #558. PR #579 closed the unknowable session failure mode: every sync /v1/runs run now records an in-flight lease (sh:run:<sid> marker + heartbeat, harness/src/run-lease.ts), and a periodic reaper in every worker finalizes orphans as failed/owner_lost so mocactl get reads a legible terminal status instead of 404/queued forever. The interrupted run's work is still lost — the reaper writes a tombstone, nothing re-executes. This issue is the design session for true recovery: making a crashed run's outcome (or the run itself) survive the death of the worker that held it.
Why the current shape can't just be extended
Three constraints any design has to answer for:
- The caller's socket dies with the worker. A sync run answers over the POST connection the spawn worker holds (
packages/mocactl/src/subagent.tsrunSpawnWorker→submitLeafTurn); the crash severs it (UND_ERR_SOCKETledger row). Even a perfect server-side resume has nobody to hand the result to. Recovery therefore implies a decoupled result-delivery contract — the client polls/runs/statusto terminal, or attaches the way detachable turns do — not just server-side re-execution. - Re-queueing the envelope is refused for authenticated callers today. The async work-queue path (
handleEnqueueLeafParsed,packages/knative-server/src/server.ts) answers 501async_runs_unavailablewhen callers authenticate, because running an owned run later would mean storing the caller's bearer in Redis (MI1 §6.5: "owned asynchronous runs are MU2's"). Any re-execution design needs an answer for credentials: short-lived re-dispatch tokens minted by the control plane, MU2's credential escrow, or an explicit "recover on the ambient credential" policy decision. - Side effects live in the sandbox, and the sandbox is released. A crashed run's sandbox lease TTL-expires back to the pool and, past the affinity grace window, the next selection for the session does a
workspaceReset(harness/src/select-sandbox.ts,SandboxAffinityPendingErrorpath). The Pi session log (transcript) survives in Redis, so conversational context is recoverable — filesystem/tool side effects since the last checkpoint are not, unless the workspace is deliberately preserved or the recovery re-executes from a checkpoint.
Building blocks that already exist
| Mechanism | Where | What it gives a design |
|---|---|---|
| Run lease + reaper | harness/src/run-lease.ts (PR #579) |
Death detection (owner_lost), atomic claim, holder identity — the trigger for recovery instead of tombstoning |
| Detachable turns | harness/src/turn-registry.ts, turn-reattach spec (docs/specs/2026-10-09-turn-reattach-design.md) |
The full reattach model: per-session lease, replayable frame log, owner_lost terminals written by an attach, cancel fencing. The closest existing precedent for "the work outlives the holder" |
| Async work queue | packages/work-queue, leaf-job.ts startup reap |
At-least-once redelivery with dead-lettering — but only for unauthenticated runs today (constraint 2) |
| Compaction checkpoints | budget-aware checkpoints in harness/ |
A resume point coarser than "re-run the whole envelope" |
| Result store | harness/src/leaf-result-store.ts |
The terminal record clients already poll (GET /runs/status) |
Open questions for the design session
- Recover what? (a) re-execute the envelope from scratch in a fresh sandbox (simplest, at-least-once), (b) resume mid-turn from the session log + last checkpoint (needs turn-level fencing), or (c) only make the result durable and re-pollable, accepting the work is lost (that is roughly #579 plus a client contract change). Is (c) enough to close the operational pain, or do spawn users need (a)?
- Exactly-once vs at-least-once. A re-executed run and a slow-reviving original holder can both reach
writeResult(plainSET). The detachable-turn design solved this with turnId fencing +termmarkers; a recovery design needs the equivalent, or an explicit "last writer wins is fine" decision with the token-spend implications stated. - Client contract.
mocactl spawn's worker currently infers failure from its severed POST and the ledger treats terminal states as authoritative without re-polling (packages/mocactl/src/subagent.ts, "Ledger is authoritative past terminal"). If recovery lands server-side, the spawn worker should follow/runs/statusto terminal instead — and theUND_ERR_SOCKETwording (#558 point 3) should stop reading like a server verdict. - Scope:
/v1/turntoo? Detachable turns already have reattach; is the right shape "make/runsdetachable underneath" (one model, two routes) or a queue-based retry that keeps the routes distinct? - MU2 dependency. If the credential answer is "control-plane-minted re-dispatch tokens," that is MU2-shaped work; decide whether this design blocks on it or ships an unauthenticated/operator-credential subset first.
Reproduction baseline
The #558 repro (delete pod -l app=moca-supervisor ~20 s after mocactl spawn --kind prompt ...) is still the right acceptance scenario; "recovered" would mean the spawned session reaches its true terminal state (or a resumed run completes) without operator intervention or re-dispatch.
- Dominant language
- TypeScript
- Stars
- 4
- Forks
- 9
- Avg merge
- 9h 31m
- Merged PRs (30d)
- 139
Getting set up
- Ships a Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from rossoctl/moca
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 64/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
Maintainers usually reply within 1 day
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
rossoctl/moca#421 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
Maintainers usually reply within 1 day
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
farion1231/cc-switch#8829 ·
Maintainers usually reply within 1 day
-
[core-client] Remove unused hand-maintained transport helpers after the definition-table cutoverOpendoc-audit enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
-
Snapshot capture failure (EPERM spawning git) fails the whole step despite the best-effort contractOpenreproduced
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
anomalyco/opencode#58685 · 1 comment ·
Maintainers usually reply within 1 day