warm pools: an attempt whose binding is missing is reaped as pod_lost while it runs, and its busy worker is offered again

Closed
  • Is something wrong?
#1,600 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

A pull request for this has already been merged.

  • #1784 by @neochaotic — merged

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
25/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
go, kubernetes, sql
Domain
backend, databases, distributed-systems

Research direction

Start with BindWarmAttempt in internal/storage/queries/runs.sql and its caller in internal/agentrpc/await_assignment.go, then onLeaseExpire in worker_registry.go, which returns the worker to the free list. The pod-lost candidate query in runs.sql is also involved. Done means the bind applies even when the ack arrives late, and a busy warm worker is not offered again or reaped as pod_lost. Reproduce with the throwaway tests described in the issue first.

Written by the indexing model from the issue text.

Description

area/executor bug priority/p0

Found in the 2026-10-09 performance and reliability re-audit (finding R-04). Warm pools only (opt-in).

What happens

  • BindWarmAttempt (internal/storage/queries/runs.sql:1186-1205) admits only queued and running; a zero-row bind returns nil (internal/storage/agent_store.go:265-283), and the bind is best effort (internal/agentrpc/await_assignment.go:176-199).
  • After the 30 s lease, onLeaseExpire puts the worker back on the free list (worker_registry.go:283-289); a later ack finds no lease and writes no binding.
  • The pod-lost candidate list is warm_worker_id IS NULL (runs.sql:1367), so an unbound warm attempt looks like a dedicated attempt whose pod vanished: it is failed as pod_lost after the 60 s grace and re-placed, and the agent kills the running user process at its next heartbeat. ListBusyWarmWorkerPods also reads the worker as idle, so the reconciler may scale it down.

Three triggers: a DB error on the bind (all modes), an ack after the 30 s lease (all modes), and in synchronous dispatch an ack that lands before the tick's MarkQueued.

Evidence

Throwaway tests on main 024196b9: after pod-lost reaper: state=failed error="pod_lost: ..." for a heartbeating warm attempt; late ack: binding=<nil> ok=false ... secondAssignedToSameBusyWorker=true.

Fix shape

Admit scheduled in the bind guard (the claim is already epoch pinned), retry the bind until it applies, and do not return a leased worker to the free list while its ack can still arrive. Later (design): make the binding part of the assignment protocol, or record warm_worker_id at claim time.

Dominant language
Go
Stars
46
Forks
1
Avg merge
11h 14m
Merged PRs (30d)
639

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from dexadata/dexaflow

All issues in dexadata/dexaflow

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.