A waiting request says why each worker cannot take it, on a gateway

Start with src/gateway/routing/pipeline.ts (runStages) and src/gateway/routing/AGENTS.md, then read ADR 0009 §5, which explains why refusals stay out of the stages. The work spans the stage files under src/gateway/routing/stages/, the walk in src/gateway/fleet-coordinator.ts, the CLI list output, and the console Waiting inspector. Done means every waiting gateway request lists one reason per known worker, and routing.test.ts still passes unchanged.

Part of #623.

Scope

On a gateway, a request waiting in the gateway's queue says, for every worker the gateway knows, why that worker cannot take it now: in simlock list --requests, GET /v1/lease-requests, status.get waiting, and one row per worker in the console's Waiting inspector. A request waiting in a worker's own queue keeps #636's one reason.

In short

Each walk of the gateway's queue already decides, worker by worker, who can take a request. It now keeps why each worker was passed over, in memory, and the request's entry lists one line per worker. The values and texts are feature #623 Shared values, rows "Waiting reason (gateway)", "Reason text" and "NO_CAPACITY detail".

flowchart LR
  W[Walk: queued request] --> P[Pick, as today]
  W --> E["explain: filter reasons, rank zero reasons (new)"]
  E --> X["Memos: cannot-serve, refused, claimed (new)"]
  X --> K["Keep one reason per worker (new)"]
  P -- attempt refused, re-queued --> E
  K --> L["list.get requests: workers[] (new)"]

Words used:

• Walk: one pass of the gateway over its queued requests, run on admission and on every worker view change.
• Known worker (here): every view in the gateway's worker registry, connected, disconnected (until retention drops it) or incompatible.

Before and after (wrk_a to wrk_d stand for worker labels; a worker with no label shows its id):

$ simlock list --requests
Request req_3: ci, ios iPhone 16, queued at 1, waiting 42s
Request req_9: local-agent on wrk_b, ios iPhone 16, queued at 1, waiting 5s

$ simlock list --requests          # after
Request req_3: ci, ios iPhone 16, queued at 1, waiting 42s
  wrk_a: drained
  wrk_b: full: running limit for iOS
  wrk_c: oS not installed
  wrk_d: busy with its own queue
Request req_9: local-agent on wrk_b, ios iPhone 16, queued at 1, waiting 5s — full: running limit for iOS

Only the first letter is lower-cased, so "OS not installed" prints as oS not installed.

Worker	Today	After
drained	nothing	Drained
connection lost, view kept	nothing	Not connected
no protocol in common	nothing	Protocol mismatch
no fitting installed runtime	nothing	OS not installed
answered NO_CAPACITY with capacityReason: "ram-budget"	nothing	Full: RAM budget
no free slot for iOS in its view	nothing	Full: running limit for iOS
a free iOS slot but no free global slot in its view	nothing	Full: running limit
passes every filter, at its RAM budget, another picked	nothing	Full: RAM budget
passes every filter, another worker picked	nothing	Not chosen this time
physical request, no free matching device on it	nothing	Every matching device is leased or absent
answered NO_CAPACITY with no detail, or a value outside the four, and no capacity check fails in its view	nothing	Full
request starting	nothing	no lines; the stage says it

If this goes wrong, an operator would see a worker missing from the list, a stale line, or a reason that blames the wrong check.

Technical spec

Facts

#	Fact	Source
F1	A filter stage answers keeps(worker, request): boolean.	src/gateway/routing/pipeline.ts:16-20
F2	runStages sorts workers by id, filters, ranks; no pick unless a rank decided.	src/gateway/routing/pipeline.ts:43-72
F3	The stage order is takes-requests, has-capabilities, can-serve, healthy, idle-queue, warm-hit, free-slot, ram-budget, free-capacity.	src/gateway/routing.ts:49-61
F4	takes-requests drops a worker not connected (disconnected or incompatible), then a drained one, then one whose capacity or catalog was not read this session; a starting one is kept with a catalog.	src/gateway/routing/stages/takes-requests.ts:12-17
F5	has-capabilities drops a worker missing any requires entry as available.	src/gateway/routing/stages/has-capabilities.ts:3-19
F6	can-serve drops a worker whose catalog does not serve the request; physical: no enrolled fit.	src/gateway/routing/stages/can-serve.ts:4-11, src/gateway/routing/request-match.ts:21, :122
F7	The fleet table tells platform, model and runtime apart with hasPlatform, then listsModel, then matchRequest, and calls takesRequests.keeps and hasCapabilities.keeps.	src/gateway/routing/serviceability.ts:47-91, src/gateway/routing/request-match.ts:141, :150
F8	healthy drops health other than running; idle-queue drops a non-empty own queue; free-slot drops a worker with no free or evictable running slot on the request's platform, or globally; ram-budget scores 0 a worker whose view says atRamBudget for the platform.	src/gateway/routing/stages/healthy.ts:6-12, idle-queue.ts:4-10, free-slot.ts:9-16, ram-budget.ts:10-12
F9	A rank answers score(worker, request): number; ranks drop no one when they abstain; with no rank deciding there is no pick. A physical request ranks only by warm-hit, which scores 0 a worker with no pickable fit.	src/gateway/routing/pipeline.ts:59-70, src/gateway/routing/stages/warm-hit.ts:6-24, src/gateway/routing/AGENTS.md "How it works"
F10	The walk removes workers with a cannot-serve memo before the table, and workers claimed this pass or refused at their current view key before select.	src/gateway/fleet-coordinator.ts:728-756
F11	Both memos are per-waiter WeakMaps; the refusal memo holds only the view key.	src/gateway/fleet-coordinator.ts:165-170, :1032-1056
F12	The view key leaves out connection and drained.	src/gateway/routing/view-state.ts:7-16
F13	NO_CAPACITY before progress remembers the refusal and walks again (noWait) or re-queues; a cannot-serve code remembers it and walks again; a grant with another lease id remembers a refusal and re-queues; an unreachable target re-queues.	src/gateway/fleet-coordinator.ts:925-946, :966-972, :828-832, :1017-1020
F14	The cannot-serve codes are UNKNOWN_MODEL, RUNTIME_MISSING, NO_DRIVER, CAPABILITY_MISSING.	src/gateway/fleet-coordinator.ts:93-98
F15	A relayed error's details are copied unchecked.	src/gateway/fleet-coordinator.ts:1228-1233, src/contract/errors.ts:453-466
F16	Every worker view change runs a walk.	src/gateway/fleet-coordinator.ts:1283-1301
F17	views() is every registry view, sorted by id; routingViews() maps it. A disconnected view is kept 24 h (gateway.disconnectedRetentionMs).	src/gateway/worker-registry.ts:147-149, :161-173, src/gateway/AGENTS.md "State and timers"
F18	The gateway's entries are requests.waiting(queue.places()); status.get waiting and list.get requests both use them; worker entries are spread from the view with workerId set.	src/gateway/fleet-coordinator.ts:208-210, src/gateway/dispatcher.ts:253, :387-397
F19	ADR 0009 §4 says the gateway "knows" a worker when it holds a catalog read from it and the worker is not incompatible.	docs/internal/adr/0009-gateway-routing-is-a-list-of-stages.md §4
F20	ADR 0009 §5: refusals "belong to one request, so they are not stages"; stages stay pure.	same ADR §5; src/gateway/routing/AGENTS.md "Rules"
F21	ADR 0021 §6: a gateway-to-worker change bumps the protocol; an older worker is incompatible; no shim.	docs/internal/adr/0021-a-gateway-dispatch-is-a-probe-and-names-its-fleet-request.md §6
F22	After #636, a worker's NO_CAPACITY carries details.capacityReason for a capacity refusal, and worker entries carry reason.	#636
F23	The CLI prints a worker entry with on <workerId>.	src/cli/index.ts:987-1010
F24	The console names a worker by label or id, and a worker it has no view of by its raw id.	ui/src/model/workers.ts:12-14, ui/src/screens/waiting/index.tsx:144-148
F26	RoutingPolicy is assess and select; routing.test.ts compares select's answer with toEqual.	src/gateway/routing.ts:29-47, src/gateway/routing.test.ts:58
F27	The CLI already reads worker.list for simlock worker list.	src/cli/index.ts:1522, src/simlock-client/client.ts:666-667
F28	The gateway e2e test lists fleet-c and the worker's local-b, one pattern per line, anchored at waiting \d+s$; its worker has label worker-a and a local request queued.	e2e/waiting-requests.test.ts:57-101
F25	Docs on the gateway list today.	docs/CLI.md:1582-1587, docs/HTTP-API.md:1562-1567, docs/CONSOLE.md:309-311

What changes

1. A filter returns its reason. FilterStage.keeps (F1) becomes check(worker, request), answering a reason to drop or undefined to keep. Each stage returns its own; the pipeline names none.

   Stage	Drops	Reason
   takes-requests	not connected: disconnected / incompatible; drained; not read this session	disconnected, incompatible, drained, not-ready
   has-capabilities	a requires entry not available	capability-missing
   can-serve	no platform; no model or class; no paired runtime; physical, no enrolled fit	no-driver, unknown-model, runtime-missing, unknown-model
   healthy	not running	unhealthy
   idle-queue	own queue not empty	busy
   free-slot	no free or evictable slot on the platform; else none globally	full + platform-running-limit; full + global-running-limit

   can-serve checks in the order of F7 with the same helpers. The fleet table (F7) calls check(...) === undefined and answers as today.

   A RankStage may also give zeroReason(worker, request): a reason for a worker it scores 0. It is generic; the pipeline names no stage. ram-budget gives full + ram-budget. warm-hit gives no-free-physical for a physical request, nothing for a virtual one.

2. A new policy method explains; select stays. RoutingPolicy gains explain(request, views), built on a pipeline function beside runStages: for each view, the first filter reason in stage order, else the first rank zero reason, else none. select and runStages keep their signatures and answers (F2, F26).

3. The coordinator keeps one reason per worker, per walk. Each walk of a waiter stores, in memory beside its other memos (F11), one entry per view in routingViews() (F17), in worker id order. The first failing check wins, in this order:

   • the filter reasons from explain, in stage order (F3), over every view, memo-excluded ones included; so drained comes before a remembered refusal (F12);
   • a cannot-serve memo (F10): UNKNOWN_MODEL → unknown-model, RUNTIME_MISSING → runtime-missing, NO_DRIVER → no-driver, CAPABILITY_MISSING → capability-missing;
   • a refusal memo at the current view key (F10): full, with the capacityReason that refusal carried, else the rank zero reason's capacity reason, else plain full;
   • claimed by an earlier request in this walk: not-chosen;
   • else the rank zero reason from explain (full + ram-budget, or no-free-physical);
   • else not-chosen: it passed every check and lost to the picked worker.
     A capacity check's full carries the capacityReason it can see: free-slot and ram-budget from the view, a remembered refusal from the worker's own answer. A refused worker that free-slot drops lists free-slot's reason: the filter comes first. Plain full remains only when none knows a reason.
     explain gives reasons only. The pick, and the views the walk hands to assess and select, are as today (F10), so ADR 0009 §5 holds: the memos still remove workers before the stages pick, and running the stages over memo-excluded views decides nothing.

4. A refusal keeps its capacity reason. The refusal memo (F11) also keeps details.capacityReason from the NO_CAPACITY answer when it is one of the four (F22), checked against the contract's enum (F15). Absent, or any other value: plain full. A grant with another lease id (F13) is plain full.

5. Back in the queue, every entry is fresh. When an attempt re-queues the request (#staleView's enqueue, F13), the coordinator runs the reason step for every view at once, from the current views and memos, so no entry is left from the walk that picked. An unreachable target (F13) gives disconnected for that worker. The next walk (F16) replaces every entry again.

6. The listing. Each queued gateway entry carries workers; a starting one none. status.get waiting and list.get requests both carry it (F18). A worker's own entries keep their reason (F22); a workers field on them is dropped.

7. CLI. Under a gateway entry with workers, one line per worker: two spaces, <workerId>: , then the text with only its first letter lower-cased. The CLI names each worker by its label, else its id, read once from worker.list when an entry has workers (F27); it decides nothing. full with a capacity reason uses that reason's text (feature #623 Shared values "Reason text"). The platform word is the request's.

8. Console. The inspector's Why shows one row per worker, labelled by the worker's name, or its raw id when the console has no view of it (F24), with the text. A single-host entry keeps #636's one row.

Modules touched

Searches: keeps(, FilterStage, RankStage, runStages, select(, RoutingPolicy, #refusals, #cannotServe, waitingRequests, #fleetWaiting, capacityReason, formatWaitingRequests, WaitingRow.

• src/contract/schemas.ts, src/contract/protocol.ts
• src/gateway/routing/pipeline.ts, src/gateway/routing/serviceability.ts, src/gateway/routing/index.ts, src/gateway/routing/stages/takes-requests.ts, has-capabilities.ts, can-serve.ts, healthy.ts, idle-queue.ts, free-slot.ts, ram-budget.ts, warm-hit.ts
• src/gateway/routing.ts, src/gateway/fleet-coordinator.ts, src/gateway/dispatcher.ts
• src/cli/index.ts
• ui/src/model/waiting.ts, ui/src/screens/waiting/index.tsx
• e2e/waiting-requests.test.ts: the gateway test's patterns gain one worker line (F28)
• Docs: docs/CLI.md, docs/HTTP-API.md, docs/CONSOLE.md; manuals src/gateway/AGENTS.md, src/gateway/routing/AGENTS.md, src/contract/AGENTS.md

flowchart LR
  cli["CLI (changed)"] --> gd["gateway dispatcher (changed)"]
  ui["console Waiting (changed)"] --> gd
  gd --> fc["fleet coordinator (changed)"]
  fc --> rt["routing policy (changed)"]
  rt --> st["stages, pipeline, table (changed)"]
  fc --> wl[worker link]
  gd --> ct["contract (changed)"]

sequenceDiagram
  participant C as Client
  participant G as Gateway
  participant W as Worker b
  C->>G: lease.request
  G->>G: walk: a drained, b picked
  G->>W: lease.request noWait (probe)
  W-->>G: NO_CAPACITY, capacityReason platform-running-limit
  G->>G: queue; b: full + platform-running-limit
  C->>G: list.get requests
  G-->>C: workers [a: drained, b: full, platform-running-limit]
  W-->>G: (or) NO_CAPACITY, capacityReason "x" → b: full

Contract and event changes

• waitingRequestSchema gains workers?: { workerId: string, reason: enum of the thirteen gateway reasons, capacityReason?: enum of the four }[]. capacityReason is accepted only with reason: "full"; no workerId twice. The array is bounded by the views the registry holds: one entry per view, no numeric limit, since the registry already bounds its views. A worker's own entries lose any workers field before they are listed.
• This PR bumps the protocol by one, min with it (F21).
• Events: none.

Other code on the same state

Searches: readers and writers of #refusals, #cannotServe, routingViews(), FilterStage (src/gateway: 4 memo methods reading or writing them, 1 routingViews() caller, 6 filter stages, the fleet table); readers of waitingRequests() (2 in src/gateway).

• Fleet table (F7): calls two filters; answers as today.
• request.dispatched: reads the pick's stage and reason; unchanged, since select and runStages are unchanged (F2, F26).
• Refusal memos (F11): read for the entries; a refusal now also keeps its capacity reason. Entries die with the waiter.
• aggregateStatus (src/gateway/aggregate.ts:85): passes waiting through.
• #636: worker entries carry reason, passed through by #fleetWaiting (F18). This task depends on #636.
• #629, #631, #635, #636 bump the protocol; the PR that merges second bumps again.

Failure modes

Call	Outcome	What happens	Limit, counted from
Probe lease.request	NO_CAPACITY before progress, valid capacityReason	full + reason, re-queue or one more walk (F13)	gateway.leaseRequestTimeoutMs (5 min) from the call, as today
Probe	NO_CAPACITY, no or bad capacityReason	plain full, unless a capacity check in the view gives one	same
Probe	cannot-serve code before progress	its reason; walk again	same
Probe	grant with another lease id	plain full; re-queue	same
Probe	unreachable before the call	disconnected; re-queue	none
Probe	failure after progress, uplink lost, timeout	request ends, as today; no entries	same
View change	worker drained, disconnected, removed	next walk (F16) replaces the entries; a removed worker has none	none

• Restart mid-step: entries are in memory; a gateway restart loses every open request (src/gateway/AGENTS.md "Known gaps"). No test needed.
• Two callers at once: a listing reads the entries the last walk stored. One thread; no test needed.

Rules in play

• architecture.md 10: "A rule is enforced in exactly one place." can-serve and the fleet table use the same helpers in the same order; the reasons come from the stages.
• architecture.md 16: "Policy is a pure function". A filter returns a value; it stores nothing.
• architecture.md 21: "A host never names an implementation." The pipeline compares no stage name; each filter returns its reason and each rank its own zero reason.
• ADR 0009 §5: "they are not stages". The coordinator, not a stage, gives the memo reasons; explain over memo-excluded views gives reasons only, never the pick.
• safety.md 10: "Anything that arrives over the wire is a claim". capacityReason is checked against the contract's enum before it is kept.
• ADR 0021 §6: "The protocol therefore moves up by one".
• testing.md 6: through FleetLeaseCoordinator with a real WorkerRegistry, and each stage through its own test.

Tests

Seam: FleetLeaseCoordinator.request and waitingRequests() over a real WorkerRegistry and a fake directory, modelled on src/gateway/fleet-coordinator.test.ts:4295. Stages in src/gateway/routing/stages/*.test.ts and routing/pipeline.test.ts.

• takes-requests returns disconnected, incompatible and drained, disconnected for a worker both disconnected and drained, and nothing for one it keeps.
• has-capabilities returns capability-missing; free-slot returns full with platform-running-limit when the platform has no free or evictable slot (also when global has none) and with global-running-limit when only global has none; each returns nothing for a worker it keeps.
• can-serve returns no-driver, unknown-model (model, class, physical) and runtime-missing, and no-driver for a worker missing all three.
• explain gives each view its first filter reason in stage order, else its first rank zero reason, else none; ram-budget gives full + ram-budget at 0, warm-hit gives no-free-physical at 0 for a physical request and nothing for a virtual one (pipeline.test.ts, stage tests). routing.test.ts and serviceability.test.ts pass with only the check rename.
• Two workers, one drained and one with no free iOS slot: the queued request lists [{ wrk_a, drained }, { wrk_b, full, platform-running-limit }], in id order.
• Every known view has one entry: a disconnected one kept by retention and an incompatible one included; a removed worker's entry is gone after the walk its removal runs.
• A worker that answered NO_CAPACITY with capacityReason platform-running-limit lists full with it; with none, with "bogus" and with 42 it lists plain full.
• A worker that answered each of UNKNOWN_MODEL, RUNTIME_MISSING, NO_DRIVER, CAPABILITY_MISSING before progress lists the matching reason.
• A worker that granted another lease id lists plain full.
• After a NO_CAPACITY re-queue and before any view change, every entry is recomputed: the picked worker lists full with its capacity reason, another worker drained since the walk lists drained, and one that passed every check but lost to the picked worker lists not-chosen.
• Entries follow the latest walk: a worker drained lists drained; after it is undrained and refuses at the RAM budget it lists full, ram-budget.
• A starting request lists no workers; a physical queued request lists one entry per worker.
• takes-requests returns not-ready for a connected worker whose capacity or catalog is not read; healthy returns unhealthy; idle-queue returns busy.
• Order: a worker drained and refused at its current view key lists drained; one refused with ram-budget and with no free platform slot lists full with platform-running-limit; one refused and otherwise passing lists full with its capacity reason.
• A worker claimed by an earlier request in the same walk lists not-chosen for the later one; a worker refused with no detail and at its RAM budget lists full with ram-budget; a physical request whose worker has no pickable device lists no-free-physical.
• A target unreachable at the attempt lists disconnected after the re-queue.
• Gateway dispatcher: list.get requests and status.get waiting carry workers on fleet entries; a worker's own entry keeps its reason, and a workers field a worker sends on it is dropped. Extend src/gateway/dispatcher.test.ts:1436.
• Contract: workers accepts each of the thirteen reasons, and capacityReason with full; it refuses an unknown reason, capacityReason with drained, an unknown capacity reason, and one workerId twice (src/contract/schemas.test.ts).
• CLI: a gateway entry prints one indented line per worker, named by label or else id, for every reason, no-driver for iOS and Android, full plain and with each capacity reason, only the first letter lower-cased; a worker entry prints #636's suffix and no indented lines. Extend src/cli/index.test.ts:6284.
• CLI and console tables: each side's table has a text for every gateway reason the contract's enum lists.
• Console model: one Why row per worker with the worker's name, the raw id for a worker with no view, the text for every reason; a single-host entry keeps one row (ui/src/model/waiting.test.ts).
• Console screen: the inspector shows the per-worker rows (ui/src/screens/waiting/waiting.test.tsx).
• E2E, gateway (e2e/waiting-requests.test.ts:57-101): the fleet-c line is followed by worker-a: busy with its own queue (its worker has local-b queued); the local-b line keeps #636's suffix.

Done when

• The tests above pass.
• On a gateway with two fake-driver workers, one drained (simlock worker drain) and the other at its iOS running limit, a waiting simlock lease shows two lines under its request in simlock list --requests, one drained, one full: running limit for iOS, and the same two rows in the console's Waiting inspector. The one-worker case runs in e2e/waiting-requests.test.ts:57-101.
• docs/CLI.md, docs/HTTP-API.md and docs/CONSOLE.md describe the per-worker lines; the routing manual says filters return reasons.

Out of scope

• A gateway's own NO_CAPACITY naming worker reasons.
• Reasons for a request in a worker's own queue (#636).
• Keeping entries across a gateway restart.
• Workers forgotten by retention or worker.remove.

Depends on

• #636

Written by an agent.