fix(scheduler): select runners by capability (CI image availability + workspace-root reachability), and retry infrastructure failures elsewhere

Start with crates/gitforge-scheduler/src/policy.rs and crates/gitforge-scheduler/src/assigner.rs to trace runner selection, then read crates/gitforge-sandbox/src/docker.rs for image availability behavior. Map how runner registration and heartbeats expose capabilities, and inspect existing scheduler contract tests. Done means tests cover image and workspace-root matching, explicit diagnostics when no runner qualifies, and retry history across different runners.

Summary

The scheduler assigns CI jobs to a runner by declared capacity only. It has no model of what that runner can actually do for the job: which CI image it holds, and whether it can reach the job's workspace root. Both gaps produce deterministic, 100%-reproducible infrastructure_failure for any repository whose CI image or workspace is not available on the runner the scheduler happens to pick.

This is a design defect, not a transient.

Code evidence

• crates/gitforge-scheduler/src/policy.rs:31-47 — SimplePolicy::select_runner filters to status == "online" && capacity > 0 and then takes max_by_key(|r| r.capacity). The comment "In a real implementation, we'd track actual load" is still true of load; there is no capability term at all.
• crates/gitforge-scheduler/src/assigner.rs:394 and :410 — the production scheduler is constructed with SimplePolicy::new().
• crates/gitforge-scheduler/src/assigner.rs:1560 — self.policy.select_runner(job_id, &available) is the whole selection step; the assignment carries the declared image but the selection never consults it.
• crates/gitforge-sandbox/src/docker.rs:361-425 — ensure_image first calls inspect_image; only if the image is not present locally does it pull. A bare reference such as dsc-ci-rust:14 therefore resolves against Docker Hub, where the repository does not exist, and the job fails with failed to pull image after 2 attempt(s): ... pull access denied for dsc-ci-rust, repository does not exist. There is no container registry in GitForge, and .gitforge.yml documents the intended contract: "The image must be pre-built on the host." Host-side presence is thus a hard precondition that the scheduler is blind to.

Measured impact

Read from the authoritative store (/var/lib/gitforge/gitforge.db, read-only) for jobs created since 2026-11-01. Totals: 1984 succeeded, 314 failed, 46 infrastructure_failure, 6 timed out, 26 skipped, 1604 cancelled.

The 46 infrastructure failures split into seven distinct classes:

Representative job ids: b3b225ed-3ac3-426b-9b4d-0ea172402048 (pull 404), 63780150-8759-4e6e-ae29-3c17b0c86d59 (workspace not visible), 33cdd917-3bf4-4a76-adf2-e791ffbfa7e6 (exit code unreadable), a167040b-ae44-406f-8038-713fd4506b88 (mount source race), dd323d44-8065-4407-9063-a0389d5e4cdc (acquisition timeout).

Note the runner split: 29 of the 46 landed on swarmone-docker and 17 on fedora-docker. A single-cause account of "the fedora runner" does not fit this data.

Two capability gaps, proven separately

1. Image availability. The job declares image: dsc-ci-rust:14. On 2026-11-04 the fedora-docker runner did not hold that tag locally, and 3 of its jobs failed with the pull 404 above. The same tag was present on swarmone-docker, which the scheduler never chose for those runs because it declares capacity: 5 against swarmone-docker's 4. After copying the identical image (all 13 layer digests match) onto the Fedora runner, the next push-triggered run of the same repository was scheduled to fedora-docker and both jobs succeeded — pipeline run e3bd1f90-7e81-47ec-a3d7-33c3ae9f8fc9, commit 32dd2db860b8589a7daf6127756e4f35528946ac, job lint 931f378e-887e-4971-aba8-73e27deeb095 and job checks c5456fda-ccd8-4a2d-8e06-5c3291941dea, both exit code 0. The only variable changed was the runner's local image store, which is exactly the capability the scheduler should have selected on.

2. Workspace-root reachability. Push-run workspaces are materialized under /nas/Temp/repos/GitForge/workspaces/<run-id> on the SWARMONE host. The fedora-docker runner reaches that root over sshfs; a directory created seconds before dispatch is not yet visible to its bind-mount preflight, so the job fails before it starts with workspace must be an existing absolute directory. All 11 occurrences fall in a 45-minute window on 2026-11-03 (21:24 to 22:10). The adjacent mkdir ...: file exists failure on the same root is the same visibility problem seen from the other side.

Expected behavior

A runner should declare what it can execute, and the scheduler should match a job against declared capabilities rather than against free capacity alone.

• Runners report, at registration and on heartbeat: the CI images they hold locally, and the workspace roots they can serve (or a boolean per root).
• The assigner selects a runner whose capabilities satisfy the job's declared image and working-directory root; if none does, the job waits or is graded with an explicit, actionable diagnostic — not an opaque infrastructure_failure.
• An infrastructure_failure that is plausibly runner-local (image, workspace visibility, container create/acquisition timeout) is retried on a different eligible runner before the run is graded failed.

Acceptance criteria

• A contract test shows that a job whose declared image exists only on runner B is assigned to runner B, and that a runner without the image is not selected.
• A contract test shows that a runner which cannot reach the job's workspace root is not selected for that job.
• When no runner satisfies the job's capabilities, the run's terminal state names the unsatisfied capability (image / workspace root) instead of a generic sandbox error.
• An infrastructure_failure is retried on a different eligible runner at least once, and the run's history records both attempts with their runners.
• The runner-selection logic is unit-tested against the two-runner case where the higher-capacity runner lacks the image.

Notes

• The 46-job breakdown above was read from the store of record, not from a watcher. Job ids and run ids are included so the numbers can be re-derived.
• This issue does not claim that the seven classes above share one cause. Only classes 1 and 2 are scheduler-capability problems; the exit-code-unreadable and container-create-timeout classes are runner-local execution faults that the retry behavior in the third bullet would also absorb.