EphemeralRunnerSet creates more EphemeralRunners than `spec.replicas` on scale up (stale cache, no create expectations)
I maintainer di solito rispondono entro 1 giorno
@yaronr ci sta già lavorando.
Dal 21/10/2026.
Valutazione
- Difficoltà
- 4/5
- Tempo stimato
- 3-5 giorni
- Idoneità per principianti
- 55/100
- Tipo di issue
- Bug
- Chiarezza
- Specificata chiaramente
- Stato di attività
- Attiva
- Stack tecnologico
- go, kubernetes
- Ambito
- devops, infrastructure
Direzione di ricerca
Start with controllers/actions.github.com/ephemeralrunnerset_controller.go, especially the child listing and scale-up logic around the cited lines 233 and 356; then read the regression test described in the issue. Reproduce the partial-cache sequence with replicas set to 2 and a new patch ID, and verify the reconciler ends with exactly 2 runners. The issue offers two fix directions, so discuss the approach with maintainers before implementing.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Descrizione
Controller version
gha-runner-scale-set-controller 0.15.0 (commit 1945b42820760eb50893c8da77fdfc42cf8dad45). Our build adds a listener-side patch only; controllers/actions.github.com is unmodified.
What happened
The listener asked for 2 runners (Calculated target runner count "assigned job"=2 decision=2, replicas=2). The EphemeralRunnerSet controller created 3. Controller log, all within the same second:
Scaling comparison current=0 desired=2
Creating new ephemeral runners (scale up) count=2 -> runners A, B
Ephemeral runner counts pending=1
Scaling comparison current=1 desired=2
Creating new ephemeral runners (scale up) count=1 -> runner C
Ephemeral runner counts pending=3
Scaling comparison current=3 desired=2
Deleting ephemeral runners (scale down) count=1
Skipping ephemeral runner since its runner ID is not recorded yet ... retryAfter=59s (x3)
Each extra runner pod requests real capacity, so the cluster autoscaler adds a node for it. The scale down cannot remove it until its runner ID is recorded. It is intermittent: an earlier run on the same day created exactly 2 for 2 jobs.
Analysis
Reconcilecounts child runners from the informer cache (ephemeralrunnerset_controller.go:233,r.ListwithMatchingFields{resourceOwnerKey}) and computescount := Spec.Replicas - scaleUpTotal(:356). Nothing records the runners this reconciler just created (no expectations as in the ReplicaSet controller), so a reconcile that runs before the cache holds all of them creates the missing ones again.- The reconcile that sees the partial cache is the one enqueued by the watch event of the first created runner (
Owns(&EphemeralRunner{}),:1203). It runs as soon as the creating reconcile returns. It reaches the scaling block only whenSpec.PatchIDis 0 or differs from the newest patch ID on the visible runners (:313), so the second ingredient is a listener patch (samereplicas, newpatchID) that lands inside that window. When the next patch comes later, the follow-up reconcile skips scaling and the count is exact: in a run that created exactly 2, the follow-up reconcile 1 s after the creates sawpending 2and the listener re-patched (patchID2, then 3) about 50 s later. - This gap existed in 0.14.2, but 0.14.2 made it narrow:
- creates were sequential (0.14.2
:380), and updateStatusalways issued a status PATCH after a create, becauseCurrentReplicaschanged and was forced (CurrentReplicas = -1 // ALWAYS update current replicas, 0.14.2:262). That round trip gave the watch time to deliver the last create beforeReconcilereturned.
- creates were sequential (0.14.2
- 0.15.0 removed both delays:
- #4673 creates in parallel (
:1000,errgroup,runnerBatchConcurrency = 8at:68), so all creates return at nearly the same time; - #4557 removed the counts from the status, so
updateStatusskips the PATCH when the phase is unchanged (:683).
Reconcilenow returns right after the lastCreateresponse, and the next reconcile lists before the watch has delivered the other runners.
- #4673 creates in parallel (
ephemeralRunnerSetMaxConcurrentReconcilesdoes not help: controller-runtime never reconciles one key concurrently. The race is between sequential reconciles of the same set.
Regression test
A fake-client test reproduces it without timing: reconcile with replicas: 2, let the cache-backed client list only the first created runner, bump spec.patchID, reconcile again. Unpatched 0.15.0 ends with 3 runners; with the first fix below it ends with 2. We are happy to open a PR with the fix and the test.
Suggested fix
Either of:
- Confirm a scale up with an uncached read before creating: when
scaleUpTotal < Spec.Replicas, list the children throughAPIReader(the reconciler already has one; filter ownership in process aspatchAppliedActionableRevisionStatusdoes, sinceresourceOwnerKeyis a cache-only index) and recompute. Cost: one namespace LIST per scale-up decision, not per reconcile. We run this as a local patch. - Add create expectations keyed by the EphemeralRunnerSet (record N on create, lower on each observed add, skip scale up while expectations are pending, with a timeout), as the ReplicaSet controller does.
Environment
Kubernetes v1.35.9 (Kamaji hosted control plane), one EphemeralRunnerSet per scale set, scale from zero.
- Lingua principale
- Go
- Stelle
- 6.7k
- Fork
- 1.5k
- Merge medio
- 1h 34m
- PR unite (30g)
- 5
Preparare l'ambiente
- Include un Dockerfile o un file Docker Compose
- Nessun modello di pull request
- Leggi la guida per i contributori
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Altre issue di actions/actions-runner-controller
-
community enhancement needs triage
Difficoltà 2/5 1-3 ore Idoneità per principianti 68/100
actions/actions-runner-controller#3997 · 9 commenti · 27 reazioni ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 4/5 3-5 giorni Idoneità per principianti 48/100
actions/actions-runner-controller#4739 · 1 commento · 4 reazioni ·
I maintainer di solito rispondono entro 1 giorno
-
gha-runner-scale-set: support Google Cloud Secret Manager as a vault for GitHub credentialsForse già presa Una pull request collegata a questa issue è aperta o già unita. Apertacommunity enhancement needs triage
Difficoltà 4/5 3-5 giorni Idoneità per principianti 52/100
actions/actions-runner-controller#4737 · 1 commento ·
I maintainer di solito rispondono entro 1 giorno
-
Controller upgrade cancels jobs that started shortly before their runner set is deletedForse di nuovo libera @ankit373 l’ha presa 9 giorni fa e non c’è nessuna pull request aperta. Aperta
Difficoltà 4/5 3-5 giorni Idoneità per principianti 45/100
actions/actions-runner-controller#4736 · 2 commenti ·
I maintainer di solito rispondono entro 1 giorno
-
EphemeralRunnerSet controller creates more runners than `spec.replicas` when its cache lags its own createsForse già presa @jia-pylon l’ha presa 14 giorni fa. Aperta
Difficoltà 4/5 3-5 giorni Idoneità per principianti 30/100
actions/actions-runner-controller#4734 · 1 commento ·
I maintainer di solito rispondono entro 1 giorno
Tutte le issue di actions/actions-runner-controller
Issue simili
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 77/100
mozilla-ai/cq#601 ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 84/100
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 78/100
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 78/100
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 1/5 1-3 ore Idoneità per principianti 78/100
I maintainer di solito rispondono entro 1 giorno