A quarantined device says what failed

Start in src/core/quarantine-coordinator.ts and src/core/reclaim-coordinator.ts to see where quarantine is entered, then src/core/registry.ts for the device record. The change also spans src/contract/schemas.ts, src/cli/index.ts and ui/src/model/attention.ts, and it depends on feature #623 and a protocol bump. Done means every quarantine path records a cause that survives restart and clears on exit.

Part of #623.

Scope

A quarantined device says what failed. simlock status, status --json, list --devices, GET /v1/status and the console's Attention inspector show the cause: the kind of failure and the tool's short error message. An agent-role caller sees the kind without the message. The cause is kept in the registry, so it is still there after a daemon restart.

In short

Every way into quarantine writes a cause on the device: purge-failed, shutdown-failed, delete-failed or stalled (with the state the device stalled in). The first three carry the error message, cut to 300 characters. The cause goes when the device leaves quarantine. The shape is feature #623 Shared values, "Quarantine cause".

flowchart LR
  R[Reclaim fails] -->|purge / shutdown / delete| Q["QuarantineCoordinator.enter (changed: writes the cause)"]
  D[doctor --fix on a stall] --> S["enterFromStalledTransition (changed: writes the cause)"]
  Q --> Reg[(state.json device record)]
  S --> Reg
  Reg --> St["status.get / list.get (changed)"]
  St --> C["Attention inspector: Cause (changed)"]
  St --> CLI["simlock status (changed)"]
  Reg -->|recover, delete, unenrol, re-add| X[cause removed]

Words used:

• Cause: why a device is quarantined, e.g. { "kind": "purge-failed", "message": "DriverCrashError: simctl erase exited 164" }.
• Message: the error text simlock already logs for the failure (<error name>: <error message>), with control characters turned into spaces. A longer text is cut so that it ends with … and is 300 characters with it. An empty one is left out.

The session, before and after:

$ simlock status
Device dev_7: quarantined, mode full, serves default mode: yes (purge retry 0, next retry at 1760619150000)

$ simlock status        # after
Device dev_7: quarantined, mode full, serves default mode: yes (purge retry 0, next retry at 1760619150000; purge failed: DriverCrashError: simctl erase exited 164)

$ simlock status --json   # after, the device entry
{ "id": "dev_7", "state": "quarantined", "quarantineAttempts": 0, "quarantineNextRetryAt": 1760619150000,
  "quarantineCause": { "kind": "purge-failed", "message": "DriverCrashError: simctl erase exited 164" }, ... }

How the device got there	Today	After: quarantineCause	Inspector Cause	simlock status marker ends with
Purge after a lease failed (virtual, or a physical device that is present)	nothing	purge-failed, message	Purge failed: <message>	purge failed: <message>
A returned physical device's wipe failed while present	nothing	purge-failed, message	Purge failed: <message>	purge failed: <message>
A spent fresh device's shutdown failed	nothing	shutdown-failed, message	Shutdown failed: <message>	shutdown failed: <message>
A spent fresh device's delete failed after the shutdown	nothing	delete-failed, message	Delete failed: <message>	delete failed: <message>
doctor --fix on a device stuck provisioning	nothing	stalled, stalledState: "provisioning"	Stalled while provisioning	stalled while provisioning
doctor --fix on a device stuck reclaiming	nothing	stalled, stalledState: "reclaiming"	Stalled while reclaiming	stalled while reclaiming
Quarantined before this upgrade (no cause in state.json)	nothing	none	no Cause line	as today: purge retry …, nothing after it

If this goes wrong, an operator sees a quarantined device with no cause or the wrong one, or a cause still showing on a device that is back in use.

Technical spec

Facts

#	Fact	Source
F1	A failed purge goes to quarantine through #recoverPurgeFailure, unless the device is physical and not present (then it goes absent).	src/core/reclaim-coordinator.ts:110-116, :322-336, :339-353
F2	A physical device's purge is the same #purge: after its lease (reclaim), and for a returned device or a deferred wipe (reclaimReturned, recoverInterrupted → #wipeInBackground).	src/core/reclaim-coordinator.ts:96-103, :146-150, :175-194, :226-231
F3	A spent fresh device's failed shutdown and its failed delete after the shutdown both hand over attemptedStrategy: "delete"; nothing tells them apart today.	src/core/reclaim-coordinator.ts:263-273, :281-291
F4	The failure handed to quarantine carries error: stableError(error). stableError returns `${name}: ${message}`.	src/core/reclaim-coordinator.ts:333; src/core/stable-error.ts:8-11
F5	QuarantineCoordinator.enter writes the entry, emits device.purge-failed only when there is a lease, then device.quarantined with deviceId, maxRetries, nextRetryAt.	src/core/quarantine-coordinator.ts:74-98
F6	enterFromStalledTransition re-checks the state and takes the reclaim claim; if either fails it writes nothing. It emits only device.quarantined.	src/core/quarantine-coordinator.ts:101-123
F7	doctor's fix passes the finding's state, which is provisioning or reclaiming.	src/core/doctor.ts:78, :137, :855; isStalledTransition :986-990
F8	Registry.enterQuarantine(deviceId, nextRetryAt) accepts a device in reclaiming, provisioning or shutdown and sets quarantineAttempts, quarantineNextRetryAt, quarantinedAt.	src/core/registry.ts:335-356
F9	A failed retry and a stranded give-up keep the device quarantined and change only the attempt fields.	src/core/registry.ts:359-377, :418-429; src/core/quarantine-coordinator.ts:298-338
F10	Leaving quarantine drops the quarantine fields in four places: recoverFromQuarantine (→ ready/shutdown), deleteQuarantined, reenrollPhysical, unenrollPhysical.	src/core/registry.ts:393-408, :431-444, :671-676, :711-715
F11	markDeviceMissing (via #commitMissing) and transitionDevice move a device out of its state without dropping the quarantine fields.	src/core/registry.ts:525-538, :622-638, :289-312
F12	Device record keys are listed once; a key not in the list is kept on disk as an unknown field and written back.	src/core/registry.ts:1083-1108, :1008-1016, :954-959
F13	A device field that is present with the wrong type fails the load (RegistryLoadError); a missing one is "a record written by a daemon that predates the field".	src/core/registry.ts:1310-1321
F14	state.json is written with one atomic write per commit.	src/core/registry.ts:949-953
F15	The dispatcher sends the device record as stored, plus derived fields; the output schema strips any field it does not declare.	src/daemon/dispatcher.ts:979-997; src/contract/schemas.ts:8-12; src/contract/AGENTS.md "Rules"
F16	status.get devices and list.get devices use statusDeviceSchema (and deviceRecordSchema for list.get's full records). Neither declares a cause.	src/contract/schemas.ts:124-151, :159-180; src/contract/operations.ts:102, :580-586
F17	A worker view, which the console and a gateway read, parses devices through statusDeviceSchema.	src/contract/worker-view.ts:69, :99; ui/src/model/workers.ts:5-6
F18	Every change to a contract shape bumps the protocol by one. It is { min: 39, max: 39 } today.	src/contract/AGENTS.md:58-60; src/contract/protocol.ts:13; src/contract/protocol.test.ts:51
F19	Precedent for a short text from a daemon: capabilityReason replaces control characters (\p{Cc}, U+2028, U+2029) with spaces, trims, cuts with slice, drops a lone high surrogate at the end; no ellipsis. The schema refuses control characters.	src/contract/schemas.ts:1056-1068, :1081-1086; docs/HTTP-API.md:275-276
F20	simlock status prints a quarantined device's marker as purge retry <n>[, next retry at <ms>].	src/cli/index.ts:2105-2118, :2166-2173; src/cli/index.test.ts:4096
F21	simlock list --devices prints JSON only.	src/cli/index.ts:979-980, :2008-2010
F22	The Attention inspector's device facts are Device, State and, when known, In this state for.	ui/src/model/attention.ts:143-155; ui/src/model/attention.test.ts:278-285
F23	The device.quarantined payload type and both catalog rows.	src/bus/index.ts:154-158; docs/EVENTS.md:45; docs/internal/EVENTS.md:83
F24	End-user docs on quarantine and status device fields.	docs/CLI.md:1404-1414; docs/HTTP-API.md:282-303; docs/CONSOLE.md:328, :337-340
F25	A daemon-level test makes the fake driver's purge fail after a lease.	src/daemon/main.test.ts:918-931
F26	At start, quarantine retries are re-armed from the stored records.	src/core/startup-converger.ts:45-48; src/core/quarantine-coordinator.ts:125-130
F27	device.quarantined is emitted in two places, both in QuarantineCoordinator; its payload has no platform. enterQuarantine returns the committed record, whose spec.platform is ios or android.	src/core/quarantine-coordinator.ts:91-95, :115-119; src/bus/index.ts:154-158; src/core/registry.ts:355; src/contract/schemas.ts:22
F28	status.get is an agent operation and list.get an admin one. A socket session is admin when it presents the host's admin secret (the host CLI's admin connection) and for an uplink, else agent; over HTTP an operator token is admin, an agent token agent. A handler already shapes its answer by role: lease.list filters by session.role. The gateway's status.get handler builds its own answer.	src/contract/operations.ts:93-95, :576; src/daemon/main.ts:406; src/daemon/server.ts:679; src/http/dispatcher-session.ts:21-25; src/daemon/dispatcher.ts:355, :561-566; src/gateway/dispatcher.ts:152, :236
F29	The console e2e test quarantines a device on the fake driver (scripted reclaim failure erase failed, thrown as a plain Error), restarts the daemon in place, then opens the Attention inspector.	e2e/console/attention.spec.ts:34-36, :88-96, :104-129, :331-374; e2e/console/fixtures.ts:41-45; e2e/fake-driver/fake-driver.ts:801
F30	A restarted daemon with a new in-process FakeDriver finds no device on its platform and marks it missing, so main.test.ts cannot carry a quarantined device across a restart.	src/daemon/main.test.ts:820-837

What changes

1. The cause. quarantineCause on the device record, as feature #623 Shared values, "Quarantine cause". stalledState only on stalled, always there on it. message on the other three kinds, never on stalled.
2. Who names the kind. ReclaimCoordinator names it on the failure it hands over (F1-F3): every #purge failure (virtual, physical after a lease, returned, deferred) is purge-failed; a failed spent shutdown is shutdown-failed; a failed delete after it is delete-failed. attemptedStrategy and device.purge-failed stay as today (F3, F5).
3. Who writes it. QuarantineCoordinator passes it to enterQuarantine, which writes it with the state in the same commit (F8, F14). enterFromStalledTransition writes stalled with stalledState = the finding's state (F6, F7). Its state parameter narrows from an optional DeviceState to "provisioning" | "reclaiming", required, the type doctor already passes (F7). Nothing else writes it: it is written once, on entry. A failed retry and a stranded device keep it (F9).
4. The message. Built once, by one function beside the schema bound, cleaned like capabilityReason (F19): control characters to spaces, then trim. Over 300 UTF-16 units: the first 299, a lone high surrogate at their end dropped, then …, so the result is at most 300. Exactly 300 or fewer: kept whole, no …. Empty after cleaning: no message. Input is the stableError text (F4). device.purge-failed's error stays the full text.
5. The event. device.quarantined gains platform, kind, stalledState?, message?, emitted after the commit as today. kind, stalledState and message are the record's values; platform is the committed record's spec.platform, as feature #623 Shared values, "Device event platform". Both emit sites carry all four (F27).
6. Gone on every exit. Any write that moves a device out of quarantined removes quarantineCause: the four of F10, and markDeviceMissing and transitionDevice (F11). One place in the registry does it.
7. Restart and old records. The cause is a known record key (F12) and is parsed at load. No cause → none shown, nothing invented. On a quarantined record, a cause that does not fit the schema fails the load like any wrong-typed field (F13). On any other record, a cause is dropped at load and never checked, malformed or not: the state rule (item 6) wins over the shape check. This covers a file an older daemon wrote back after a rollback (F12).
8. On the wire. deviceRecordSchema and statusDeviceSchema declare quarantineCause (F15-F17). This PR bumps the protocol by one (F18).
9. CLI. simlock status puts the cause last in the quarantine marker, so commas in the message cannot be read as marker parts: today's marker, then ; , the kind text lower-case, then : <message> when there is one (F20). Example: (purge retry 0, next retry at 1760619150000; purge failed: DriverCrashError: simctl erase exited 164). No cause: today's marker alone. list --devices and --json carry the field as sent (F21).
10. By role. For an agent session, status.get leaves message out of every device's cause, in devices and in each workers[].devices; kind and stalledState stay. An admin session sees it all. One function beside the cause schema does the strip; the worker's and the gateway's status.get handlers both call it (F28). list.get is admin only, so it always carries the message.
11. Known gap, written into the core manual's Known gaps: after a rollback, an older daemon keeps the cause as an unknown field (F12). If the device leaves quarantine and enters it again under that older daemon, a newer daemon loads a quarantined record whose cause is from the earlier entry. It cannot be told apart, and is shown as is.
12. Console. The Attention inspector of a quarantined device adds a last fact Cause: the kind text, then : <message> when there is one. Kind texts: Purge failed, Shutdown failed, Delete failed, Stalled while provisioning, Stalled while reclaiming. No cause: no Cause fact (F22).

Modules touched

Searches: quarantinedAt|quarantineAttempts|quarantineNextRetryAt, enterQuarantine|enterFromStalledTransition|quarantine.enter, "device.quarantined", statusDeviceSchema|deviceRecordSchema, quarantineMarker, deviceFacts.

• src/core/reclaim-coordinator.ts, src/core/quarantine-coordinator.ts, src/core/registry.ts, src/core/domain.ts (record type), src/core/AGENTS.md (step 4).
• src/bus/index.ts (payload type).
• src/daemon/dispatcher.ts, src/gateway/dispatcher.ts (status.get by role).
• src/contract/schemas.ts (cause schema, message function), src/contract/protocol.ts, src/contract/AGENTS.md (protocol number).
• src/cli/index.ts.
• ui/src/model/attention.ts.
• e2e/console/attention.spec.ts.
• docs/CLI.md, docs/HTTP-API.md, docs/CONSOLE.md, docs/EVENTS.md, docs/internal/EVENTS.md.

flowchart LR
  RC["ReclaimCoordinator (changed)"] --> QC["QuarantineCoordinator (changed)"]
  DOC[Doctor] --> QC
  QC --> REG["Registry (changed)"]
  QC --> BUS["bus: device.quarantined (changed)"]
  DISP["DaemonDispatcher (changed)"] --> REG
  DISP --> SCH["contract schemas (changed)"]
  GW["GatewayDispatcher (changed)"] --> SCH
  CLI["CLI status (changed)"] --> DISP
  UI["console attention model (changed)"] --> SCH

Contract and event changes

• deviceRecordSchema, statusDeviceSchema: quarantineCause?: { kind: "purge-failed" | "shutdown-failed" | "delete-failed" | "stalled", stalledState?: "provisioning" | "reclaiming", message?: string }. message: 1-300 characters, no control characters. stalledState present exactly when kind is stalled; message never on stalled. A bad value fails the parse: in a daemon's own output it is an output check failure, so the message function (item 4) keeps every written value inside the bound. This PR bumps the protocol by one.
• status.get output for an agent session: no quarantineCause.message (item 10).
• device.quarantined: additive platform: "ios" | "android", kind, stalledState?, message? (events rule 6). docs/EVENTS.md and docs/internal/EVENTS.md rows name them.
• state.json device record: quarantineCause, same shape, checked at load (item 7).

Other code on the same state

Searches, outside tests, over src/ and ui/src/: the quarantine fields 39 hits, the entry methods 8, "device.quarantined" 3, the two device schemas 10, quarantineMarker 2, deviceFacts 2.

Component	Both act	Who wins
QuarantineCoordinator retries (F9)	A retry runs under the reclaim claim after the entry commit	Entry is first; a retry never writes the cause
PhysicalEnrollment re-add or remove of a quarantined phone (reenrollPhysical, unenrollPhysical)	Serialized with the retry by the claim	Whichever exits quarantine removes the cause
StartupConverger re-arm (F26)	Reads state and retry time, not the cause	No conflict
DaemonDispatcher (F15), worker view and gateway (F17)	Read the stored record	They show what was last committed
#630 projections	Read device.quarantined, its platform for the device fact row	#630 depends on #635 for platform; an event from before #635 has none
#318 / PR #324 (draft): device shapes gain when the device entered its state	Same schemas, registry keys and load code	They run at once; the PR that merges second takes the base branch and bumps the protocol again
#319 (draft): console time in state	Same deviceFacts	The second to merge takes the base branch; Cause stays the last fact

Failure modes

Call	Outcome	What happens	Limit, counted from
Registry.enterQuarantine with the cause	ok	State, attempt fields and cause in one write; then the event	as today
Registry.enterQuarantine	throws (write fails)	Nothing written, no event, as today	as today
enterFromStalledTransition	device claimed or moved on (F6)	Nothing written, no cause	none
Registry.load	cause missing	Loads with no cause	none
Registry.load	cause malformed on a quarantined record	RegistryLoadError, as any wrong-typed field (F13)	none
Registry.load	cause, well-formed or not, on a record not quarantined	Loaded without it	none
Gateway or console reading a worker's devices	cause malformed	Refused by statusDeviceSchema, as any bad device field today (F17). No new test: the schema line proves the refusal	as today

• Restart mid-step: one atomic write (F14), so a record has the state and cause together, or neither.
• Two callers at once: entries run inside decisions.run; enterQuarantine refuses a device already quarantined (F8), so a second entry never overwrites a cause.

Rules in play

• events rule 6, "additive changes only": four new fields on device.quarantined; nothing removed.
• events rule 8: both EVENTS.md rows.
• architecture rule 10, "enforced in exactly one place": one message function; one registry place removes the cause.
• architecture rule 12, "exactly one named state": every exit from quarantined leaves no cause.
• architecture rule 10 again: one strip function for the role, called by both status.get handlers.
• safety rule 10, "a claim, not a fact": the cause a gateway reads from a worker is bounded by the schema.
• documentation rule 3: nothing printed names a repo path; the message is the error text only.
• testing rule 7: each path's kind is proved once, at ReclaimCoordinator; the record and event once, at QuarantineCoordinator.

Tests

Seam: ReclaimCoordinator and QuarantineCoordinator (with createCore's real registry) through their public methods; model on src/core/reclaim-coordinator.test.ts:345 and src/core/quarantine-coordinator.test.ts:179.

• reclaim-coordinator.test.ts: every failed purge is handed to quarantine as purge-failed with the stableError text: a virtual device's (extend :345), a deferred wipe's (extend :430), a returned physical device's while present (extend :726), a physical device's after its lease.
• reclaim-coordinator.test.ts: a spent fresh device's failed shutdown is handed over as shutdown-failed (extend :548), and its failed delete after the shutdown as delete-failed (extend :529).
• quarantine-coordinator.test.ts: enter writes quarantineCause { kind, message } on the record and the same kind and message on device.quarantined, for purge-failed, shutdown-failed and delete-failed (extend :179).
• quarantine-coordinator.test.ts: the message replaces control characters with spaces and is trimmed; an error of exactly 300 characters is kept whole with no …; one of 301 gives its first 299 and … (300 in all); a high surrogate at character 299 is dropped before the …; one that is only whitespace or control characters gives a cause with no message; device.purge-failed's error stays the full text.
• quarantine-coordinator.test.ts: enterFromStalledTransition writes { kind: "stalled", stalledState } with no message, for a device stalled provisioning and one stalled reclaiming, virtual and physical; the event carries kind and stalledState and no message (extend :240, :744).
• quarantine-coordinator.test.ts: device.quarantined carries the device's platform from both emit sites: enter and enterFromStalledTransition, for an iOS and an Android device (extend :179, :240).
• quarantine-coordinator.test.ts: a stalled entry that writes nothing (claimed, or state moved on) leaves no cause (extend :1356).
• quarantine-coordinator.test.ts: a failed reclaim retry, a failed delete retry, a not-present wait and a stranded give-up keep the cause as written at entry (extend :319, :432, :1219, :361).
• registry.test.ts: every exit from quarantined removes quarantineCause: recoverFromQuarantine to ready and to shutdown, deleteQuarantined, reenrollPhysical, unenrollPhysical, markDeviceMissing, transitionDevice (extend :784, :833).
• registry.test.ts: a quarantined record with each kind, written to state.json and loaded by a new Registry, has its cause, with the clock short of quarantineNextRetryAt (model on :958).
• registry.test.ts: a record with no quarantineCause loads with none; a record not quarantined loads without its cause, both with a well-formed cause and with a malformed one (no RegistryLoadError); a quarantined record with a cause from an earlier entry loads as is.
• registry.test.ts: a record whose cause has an unknown kind, stalled without stalledState, stalledState on another kind, a message on stalled, a non-string, empty or 301-character message, or a control character on a quarantined record fails the load with RegistryLoadError.
• src/contract/schemas.test.ts: statusDeviceSchema and deviceRecordSchema accept each of the four kinds as Contract and event changes says, and refuse each bad case of the line above (model on :235).
• src/contract/worker-view.test.ts: a worker view keeps a device's quarantineCause.
• src/contract/protocol.test.ts:51: the range is one higher.
• src/daemon/main.test.ts: after a failed purge on the fake driver, status.get and list.get devices show quarantineCause { kind: "purge-failed", message: "DriverCrashError: purge exploded" } (model on :918). Restart is proved by the registry line and the e2e line, not here (F30).
• src/daemon/main.test.ts and src/http/app.test.ts: a quarantined device's cause has message for an admin session over the socket and an operator token over HTTP, and has kind without message for an agent session over the socket and an agent token over HTTP.
• src/gateway/dispatcher.test.ts: a gateway's status.get gives an agent session a worker's quarantined device with kind and no message, in devices and in workers[].devices, and an admin session the message.
• src/cli/index.test.ts: simlock status ends the marker with ; and the cause for each kind ((purge retry 1, next retry at <ms>; purge failed: <message>) with a message that holds a comma, shutdown failed: <message>, delete failed: <message>, stalled while provisioning, stalled while reclaiming), and a quarantined device with no cause prints (purge retry 1) as today (extend :4096).
• ui/src/model/attention.test.ts: a quarantined device's facts end with Cause for each kind: Purge failed: <message>, Shutdown failed: <message>, Delete failed: <message>, Stalled while provisioning, Stalled while reclaiming.
• e2e/console/attention.spec.ts:331: the quarantined device's inspector shows Cause Purge failed: Error: erase failed, after the daemon's restart in place (:364), when the inspector opens (:370-374).
• ui/src/model/attention.test.ts: a quarantined device with no cause has no Cause fact (extend :278), and a stalled device that is not quarantined has none.

Done when

• The Tests lines above pass.
• On the fake driver, a device quarantined after a failed purge shows Purge failed: <message> in the Attention inspector and quarantineCause in simlock status --json, before and after simlock daemon stop and start (the e2e line).
• src/core/AGENTS.md Known gaps has the rollback gap (item 11).
• docs/CLI.md (quarantine paragraph and status marker), docs/HTTP-API.md (devices fields, and no message for an agent token), docs/CONSOLE.md (quarantined row and the inspector paragraph), docs/EVENTS.md and docs/internal/EVENTS.md (device.quarantined, with platform) describe the cause and the new payload fields.

Out of scope

• Changing the cause on a failed retry, or keeping a history of causes.
• A cause for device.quarantine-stranded or -abandoned (they keep their own error).
• Re-wording today's purge retry marker for a delete retry.

Depends on

Written by an agent.