solver: build fails with "failed to get state for index" after another build sharing its steps is canceled

Start with the proposed fix in linked PR #7265 and trace how loadUnlocked adds jobs to states and how setEdge propagates them along owner chains. Read the reproducer and scheduler trace in the linked gist to follow the cancellation sequence. Done means the focused regression test passes and the reproducer no longer fails after canceling A and B.

Contributing guidelines and issue reporting guide

• I've read the contributing guidelines and wholeheartedly agree. I've also read the issue reporting guide.

Well-formed report checklist

• I have found a bug that the documentation does not mention anything about my problem
• I have found a bug that there are no open or closed issues that are related to my problem (related closed issues are linked below, with why their fixes do not cover this case)
• I have provided version/information about my environment and done my best to provide a reproducer

Description of bug

Bug description

Three builds share a chain of steps. Two of them are canceled, and the third fails with:

failed to compute cache key: failed to get state for index 0 on <vertex>

or, on the getEdge path, failed to get edge: inconsistent graph state. The third build should finish: its own job still references every step it uses.

The sequence, with scheduler debug names:

1. Job A loads chain X. Job B loads an equivalent chain X' (same cache keys, different digests), and X' edges merge into X. setEdge copies B's job onto X's states.
2. Job C loads X' by its exact digests after the merge (reusing active vertex). loadUnlocked adds C to the X' states only. X's states never receive C.
3. A and B are canceled and discarded. The X' states survive because C holds them. The X states have no jobs left, and deleteIfUnreferenced removes them with their inputs.
4. C's X' edges are owned by X's released edges. The next cache key computation looks up an X input in actives, which is gone.

On #2303, closed 2026-09-14, the request was a new issue with BUILDKIT_SCHEDULER_DEBUG=1 daemon logs for anyone still hitting this on v0.14 or later; this reproduces on v0.34.0. The fixes listed there, including #4887 (no merges into inactive states), cover merges themselves. Here the merge is valid when it happens; the defect is a job that attaches to the merge source afterwards.

Reproduction

A client driver and scripts that run the sequence against any daemon image: https://gist.github.com/c-p-b/84857d8967147d400e045394231f55fc

git clone https://gist.github.com/c-p-b/84857d8967147d400e045394231f55fc.git repro
cd repro && ./run.sh moby/buildkit:v0.34.0

run.sh starts a disposable daemon with BUILDKIT_SCHEDULER_DEBUG=1, runs 20 iterations, then runs the same sequence with nothing canceled as a control. Each iteration: a warm build whose RUN results are pruned (cache keys kept, as build-cache GC leaves them), then A builds a three-step RUN chain, B starts 300 ms later with an equivalent chain under a different local name, C starts 300 ms after B with B's exact digests, and A and B are canceled at 1.8 s.

C builds that failed:

Daemon	macOS, OrbStack, linux/arm64	Linux, EC2 Graviton
moby/buildkit:v0.34.0, A and B canceled	20 of 20	20 of 20
moby/buildkit:v0.34.0, nothing canceled	0 of 20	0 of 20
master 325ea68ea, A and B canceled	18 of 20	20 of 20

A rerun on 2026-10-11 with 3 iterations on moby/buildkit:v0.34.0 failed 3 of 3, and 0 of 3 with nothing canceled. The daemon log of that run up to the first failure is daemon-v0.34.0-first-failure.log in the gist, and ./trace.sh daemon-v0.34.0-first-failure.log prints the scheduler lines for the chain:

17:59:08 msg="adding active vertex" key="dc77e463" job=piaom41gh7fmqjb7scsdazmkf vertex_name="s0"
17:59:08 msg="adding active vertex" key="cf14b79c" job=piaom41gh7fmqjb7scsdazmkf vertex_name="s1"
17:59:08 msg="adding active vertex" key="64ae65c5" job=piaom41gh7fmqjb7scsdazmkf vertex_name="s2"
17:59:09 msg="adding active vertex" key="d75a7a49" job=u1fp794ijaf5if44vji5alkjz vertex_name="s0"
17:59:09 msg="adding active vertex" key="6564d5f8" job=u1fp794ijaf5if44vji5alkjz vertex_name="s1"
17:59:09 msg="adding active vertex" key="c5839531" job=u1fp794ijaf5if44vji5alkjz vertex_name="s2"
17:59:09 msg="merging edges" dest_vertex_digest="dc77e463" dest_vertex_name="s0" source_edge_vertex_digest="d75a7a49"
17:59:09 msg="merging edges" dest_vertex_digest="cf14b79c" dest_vertex_name="s1" source_edge_vertex_digest="6564d5f8"
17:59:09 msg="merging edges" dest_vertex_digest="64ae65c5" dest_vertex_name="s2" source_edge_vertex_digest="c5839531"
17:59:09 msg="reusing active vertex" key="d75a7a49" job=jutoesuj3x7xyji7rgx7oy0wj vertex_name="s0"
17:59:09 msg="reusing active vertex" key="6564d5f8" job=jutoesuj3x7xyji7rgx7oy0wj vertex_name="s1"
17:59:09 msg="reusing active vertex" key="c5839531" job=jutoesuj3x7xyji7rgx7oy0wj vertex_name="s2"
...
17:59:11 msg="deleting job from state" key="64ae65c5" job=piaom41gh7fmqjb7scsdazmkf vertex_name="s2"
17:59:11 msg="deleting unreferenced active vertex" key="64ae65c5" vertex_name="s2"
17:59:11 msg="deleting job from state" key="dc77e463" job=piaom41gh7fmqjb7scsdazmkf vertex_name="s0"
17:59:11 msg="deleting unreferenced active vertex" key="dc77e463" vertex_name="s0"
17:59:11 msg="deleting job from state" key="cf14b79c" job=piaom41gh7fmqjb7scsdazmkf vertex_name="s1"
17:59:11 msg="deleting unreferenced active vertex" key="cf14b79c" vertex_name="s1"
17:59:12 msg="/moby.buildkit.v1.frontend.LLBBridge/ReadDir returned error: rpc error: code = Unknown desc = failed to compute cache key: failed to get state for index 0 on s2"

A is piaom…, B is u1fp…, C is jutoe…. C's job is on B's states (d75a7a49, 6564d5f8, c5839531) and never on A's, so A's states are deleted while C's edges still route through them.

Proposed fix

#7265: when loadUnlocked adds a job to a state, also add it to every state along the owner chains of that state's edges, as setEdge already does for the jobs a state has at merge time. That keeps the existing rule that a state owning a merged edge holds every job routed through it. It includes TestLoadOntoMergedEdgeSurvivesOwnerDiscard, which fails on master and passes with the change; with the change the reproducer fails 0 of 20 on both platforms, and every active vertex added was deleted again by the end of the run (399 of 399), so the extra references do not keep states alive.

Alternatives considered, none implemented:

• Load C onto the owner state X instead of X'. C asked for X' by digest, and the state, vertex name and progress it reports would change.
• Have deleteIfUnreferenced keep a state while any active state's edge is owned by it. That needs a reverse owner index or a scan of actives on every delete, and ties state lifetime to edges while the rest of the solver ties it to jobs.
• Undo the merge when the owner's jobs are discarded. The merged edges may already hold the owner's cache keys and results, so they would have to be rebuilt.

Happy to rework the PR if a different approach is preferred.

Version information

buildkitd github.com/moby/buildkit v0.34.0 f5b59a1f2fb25a0cb7072c398cab9aed7835da4e
Docker Engine 29.4.0 (linux/arm64), OrbStack on macOS
Also reproduced on Linux 6.18, EC2 Graviton, with the same image's buildkitd binary