warm pools: assignment streams survive a leader step-down, so the pool is dark until worker_idle_ttl
Maintainers usually reply within 1 day
A pull request for this has already been merged.
- #1793 by @neochaotic — merged
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 45/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Active
- Tech stack
- go, grpc
- Domain
- backend, distributed-systems
Research direction
Start in internal/agentrpc/await_assignment.go, where the leader gate is checked once when the AwaitAssignment stream opens (around line 66) and the loop exits only on ctx, shutdown, recv or send errors (lines 95-123). Then read internal/agent/warm.go around lines 200-215 to see how the worker reacts to FailedPrecondition and re-dials. The reconcile and GC ticks are in cmd/dexaflow-server/main.go around lines 2178-2184 and 2296. Done means that a step-down closes open assignment streams and the warm tick stops, verified by a test that checks the worker is no longer registered on the follower.
Written by the indexing model from the issue text.
Description
Found in the 2026-10-09 performance and reliability re-audit (finding R-18). Warm pools only (opt-in).
What happens
AwaitAssignment checks the leader gate once, when the stream opens (internal/agentrpc/await_assignment.go:66); nothing ends the stream on step-down (:95-123 exit only on ctx, shutdown, recv or send error).
After a step-down without a restart (DB blip, lock lost, check timeout) every warm worker stays registered on the old leader, where no placer runs. The new leader's registry is empty, so every task goes to a dedicated pod, while its warm reconciler sees the old workers as live idle pods that satisfy min_idle and creates nothing. The pool is dark until worker_idle_ttl (5 minutes by default) recycles the workers.
Evidence
Throwaway test on main 024196b9: 2s after step-down: stream still open, worker still registered on the follower=true.
Fix shape
On step-down, close every AwaitAssignment stream with FailedPrecondition ("not the leader, reconnect"); the worker already re-dials toward the leader on that status (internal/agent/warm.go:200-215). Also wrap the warm reconcile and staging GC ticks in a timeout derived from the leader's run context (cmd/dexaflow-server/main.go:2178-2184, 2296), so a tick in progress at step-down stops too.
- Dominant language
- Go
- Stars
- 46
- Forks
- 1
- Avg merge
- 13h 26m
- Merged PRs (30d)
- 627
Getting set up
- Ships a Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from dexadata/dexaflow
-
area/docs documentation priority/p3
Difficulty 1/5 1-3 hours Newbie friendliness 82/100
Maintainers usually reply within 1 day
-
area/ci enhancement priority/p3
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
Maintainers usually reply within 1 day
-
area/docs documentation priority/p2
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day
-
area/docs documentation priority/p2
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day
-
area/cli bug priority/p2
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day
All issues in dexadata/dexaflow
Similar issues
-
documentation ready
Difficulty 1/5 Under an hour Newbie friendliness 88/100
Maintainers usually reply within 1 day
-
Needs Triage
Difficulty 1/5 Under an hour Newbie friendliness 90/100
-
bug good first issue
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
solongate/agent-security#154 ·
Maintainers usually reply within 1 day
-
tree: RootStyle is applied twice to the root nodePossibly taken @PhotoDotPi claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
charmbracelet/lipgloss#772 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 76/100
Maintainers usually reply within 1 day