[FEATURE] Stable FleetNode identity via StatefulSet agent pools (follow-up to #1640)
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 35/100
- Issue type
- Feature
- Clarity
- Clearly specified
- Activity status
- Active
- Tech stack
- go, helm, kubernetes
- Domain
- backend, infrastructure
Research direction
Start by reading the #1640 decision, then inspect the agent pool chart and controller paths that create FleetNodes, derive their names, recover orphaned tasks, and reap stale nodes. Compare the proposed StatefulSet behavior with the #1647 rollout settings and the #1523 selector-adoption issue; consider the #1635 supervision budget and #1634 placement spread. Done when the evaluation documents behavior, costs, and a recommendation for whether to migrate.
Written by the indexing model from the issue text.
Description
Feature Description
Evaluate moving the in-cluster foreman-agent pools from Deployments to
StatefulSets, so FleetNode identity is stable across rollouts
(foreman-coder-agent-0) instead of minted fresh from each pod's random suffix.
This is the deliberate follow-up to the #1640 decision. That decision settled
what a FleetNode identifies (the agent process, not the machine). This issue
is about how stable that identity should be — orthogonal, and deliberately not
folded into #1640 so it gets its own evaluation rather than riding along.
Problem Statement
As an operator, I want an agent pool's FleetNodes to survive a routine image
bump, so a rollout does not mint a new fleet and orphan the old one, and so
per-node state has somewhere durable to live.
Because identity is the pod name today, every rollout mints a new generation
of FleetNodes and abandons the previous one. Observed live on 2026-08-22: a
chart upgrade left eight FleetNodes in Draining until the orphan reaper
deleted them. The reaper (#979/#980) exists largely to clean up after this
churn.
Churn also has a second-order cost: it makes FleetNode.status a poor home for
any durable per-node state. That is exactly where the supervision-budget
follow-up from #1635 wants to put its bound ("a process-local int is a strange
place for capacity to live when the cluster's model is a single currentTask
string") — and with identities that die on every rollout, moving the budget to
status would just lose it on every upgrade.
Proposed Solution
StatefulSet per agent pool, ordinal-named pods, --fleet-node-name derived
from the stable pod name (which os.Hostname() already yields; the change is
that the name stops churning).
Expected wins, to be validated rather than assumed:
- Rollouts reuse FleetNodes: no Draining generation, no reaping, no window
wherekubectl get fleetnodesshows double the fleet. FleetNode.statusbecomes viable for durable per-node state
(supervision budget from #1635; anything else that wants a per-node ledger).- First-fit-alphabetical scheduling (
reserveFirstFitNode, deterministic by
name) becomes genuinely deterministic across rollouts instead of
random-suffix roulette. Note this cuts both ways: it also concentrates load
on ordinal 0 more predictably, which intersects with the placement-spread
discussion in #1634.
Costs and open questions to evaluate
- Chart surgery: per-pool StatefulSet, and how
updateStrategyinteracts with
the rollout-strategy values just added in #1647 (StatefulSets have
RollingUpdate/OnDelete, notRecreate; the #1523 selector-adoption trap
needs re-checking in the new shape). recoverOrphanedTaskssemantics when a restarted pod comes back with the
SAME FleetNode name: today a new pod resets the old node's Running tasks via
the claim-expiry path; with a stable name the restarted process inherits its
own previous claims. That may be strictly better (it can resume-or-reset
deliberately) but it is a behaviour change to reason through.- Ordered rollout semantics and whether
podManagementPolicy: Parallelis
wanted. - Whether the Draining reaper and any churn-shaped assumptions elsewhere can be
simplified once identities stop dying.
Alternatives Considered
- Stay on Deployments (the #1640 (b) baseline). Works today; the reaper
absorbs the churn. This issue exists because the churn's second-order cost
(no durable per-node state) blocks a direction #1635 already wants. - DaemonSet (one agent per Kubernetes node). Rejected in #1640: it makes
node identity coherent by destroying the role-pool model, and caps
concurrency by node count rather than workload.
Trigger
Evaluate this when the supervision-budget-in-status question from #1635 becomes
live, or earlier if FleetNode churn starts biting operationally. Filed now so
the option does not get lost between here and there.
Priority
- Medium - Nice to have
Willingness to Contribute
- Yes, I can submit a PR
- Dominant language
- Go
- Stars
- 232
- Forks
- 37
- Avg merge
- 21h 42m
- Merged PRs (30d)
- 244
Getting set up
Starts the project's dev container in your browser, under your own GitHub account.
- Ships a Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from defilantech/LLMKube
-
component/metal-agent documentation
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
defilantech/LLMKube#2581 ·
Maintainers usually reply within 1 day
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
defilantech/LLMKube#2578 ·
Maintainers usually reply within 1 day
-
documentation helm priority/low size/small
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
defilantech/LLMKube#2420 ·
Maintainers usually reply within 1 day
-
component/metal-agent enhancement priority/low size/medium
Difficulty 4/5 3-5 days Newbie friendliness 48/100
defilantech/LLMKube#2597 ·
Maintainers usually reply within 1 day
-
[BUG] Docs and comments describe InsufficientMemory only as a start refusal, but a serving engine can now carry itPossibly taken @defilan-agent claimed this today. Openbug component/metal-agent documentation priority/low size/small
Difficulty 2/5 1-3 hours Newbie friendliness 25/100
defilantech/LLMKube#2596 ·
Maintainers usually reply within 1 day
All issues in defilantech/LLMKube
Similar issues
-
☢️ Bug v3
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
Maintainers usually reply within 1 day
-
area/import-export area/projects concern/ux
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
go-vikunja/vikunja#4455 · 1 comment ·
Maintainers usually reply within 2 days
-
Dependabot PRs no longer merge automaticallyPossibly taken @dwisiswant0 claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
projectdiscovery/urlfinder#160 ·
Maintainers usually reply within 1 day
-
database
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
chnm/legal-modernism#450 ·
Maintainers usually reply within 1 day
-
tech-debt
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
Maintainers usually reply within 1 day