EphemeralRunnerSet controller creates more runners than `spec.replicas` when its cache lags its own creates
I maintainer di solito rispondono entro 1 giorno
@jia-pylon ci sta già lavorando.
Dal 4/11/2026.
Valutazione
- Difficoltà
- 4/5
- Tempo stimato
- 3-5 giorni
- Idoneità per principianti
- 30/100
Direzione di ricerca
Start with EphemeralRunnerSetReconciler.Reconcile in the controller package and trace how it lists cached child runners before creating or deleting them. Run the envtest suite with the documented command and inspect the scale-up logs for counts that exceed spec.replicas. Done means reconciliation accounts for in-flight creates and deletes, with regression tests covering the cache lag; a linked pull request is already open.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Descrizione
Checks
- I've already read https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/troubleshooting-actions-runner-controller-errors and I'm sure my issue is not covered in the troubleshooting guide.
- I am using charts that are officially provided
Controller Version
gha-runner-scale-set-0.14.2; also reproduced on master 89f34c4f782d
Deployment Method
Helm
Checks
- This isn't a question or user support case (For Q&A and community support, go to Discussions).
- I've read the Changelog before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes
To Reproduce
- Check out master and run the controller envtest suite against any recent kube-apiserver (reproduced on 1.35.0, 1.36.2 and 1.37.0 from
setup-envtest):
KUBEBUILDER_ASSETS=... go test ./controllers/actions.github.com/ -count=1 -ginkgo.v - Find a scale-up from 0 in the controller log, e.g. in
It should create/add all required resources for a new EphemeralRunnerSet (finalizer).
Describe the bug
EphemeralRunnerSetReconciler.Reconcile lists child EphemeralRunners from the informer cache (MatchingFields{resourceOwnerKey: ...}) and creates desired - current runners. Nothing tracks creates or deletes that are still in flight. Its own status patch, or the add event for the first runner in a batch, can wake the next reconcile before the cache has seen the whole batch. That reconcile counts too few runners and creates another batch. It then deletes the surplus, so specs that wait for the final count still pass, but spec.replicas has been exceeded in the meantime. Scale-down has the same shape: a cache that hasn't caught up with deletes can lead to choosing more runners to delete.
Over-creates in unmodified full-suite runs on master (each run executes all 188 specs):
| kube-apiserver | full runs | runs with an over-create in the log |
|---|---|---|
| 1.35.0 | 8 | 6 |
| 1.36.2 | 8 | 8 |
| 1.37.0 | 8 | 7 |
The race sometimes surfaces as an assertion failure instead, e.g. Should replace finished ephemeral runners with new ones on 0.14.2: current: 0, desired: 2 → 2 created → status count 0 → current: 0, desired: 2 → 2 more created → 4 pending.
Describe the expected behavior
The controller never has more than spec.replicas runners because of its own stale reads, and never deletes more than the scale-down requires. kube-controller-manager's ReplicaSet controller solves the same problem with controller expectations.
Additional Context
In production, runner sets briefly over-provision runner pods. The extra runners are deleted again, but in the meantime they use node capacity.
Controller Logs
1.36.2 envtest, one namespace, all within 2026-11-04T20:23:49Z:
Scaling comparison: current=0, deleting=0, desired=5
Creating new ephemeral runners (scale up): count=5
Created new ephemeral runner (x5, distinct names)
Scaling comparison: current=1, deleting=0, desired=5
Creating new ephemeral runners (scale up): count=4
Created new ephemeral runner (x4, distinct names)
Scaling comparison: current=7, deleting=0, desired=5
Deleting ephemeral runners (scale down): count=2
Nine runners were created where five were desired. Full logs are available on request.
Runner Pod Logs
N/A (controller-side race)
I have a fix with regression tests ready and will open a PR that references this issue.
- Lingua principale
- Go
- Stelle
- 6.7k
- Fork
- 1.5k
- Merge medio
- 1h 34m
- PR unite (30g)
- 5
Preparare l'ambiente
- Include un Dockerfile o un file Docker Compose
- Nessun modello di pull request
- Leggi la guida per i contributori
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Altre issue di actions/actions-runner-controller
-
community enhancement needs triage
Difficoltà 2/5 1-3 ore Idoneità per principianti 68/100
actions/actions-runner-controller#3997 · 9 commenti · 27 reazioni ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 4/5 3-5 giorni Idoneità per principianti 48/100
actions/actions-runner-controller#4739 · 1 commento · 4 reazioni ·
I maintainer di solito rispondono entro 1 giorno
-
gha-runner-scale-set: support Google Cloud Secret Manager as a vault for GitHub credentialsForse già presa Una pull request collegata a questa issue è aperta o già unita. Apertacommunity enhancement needs triage
Difficoltà 4/5 3-5 giorni Idoneità per principianti 52/100
actions/actions-runner-controller#4737 · 1 commento ·
I maintainer di solito rispondono entro 1 giorno
-
Controller upgrade cancels jobs that started shortly before their runner set is deletedForse di nuovo libera @ankit373 l’ha presa 9 giorni fa e non c’è nessuna pull request aperta. Aperta
Difficoltà 4/5 3-5 giorni Idoneità per principianti 45/100
actions/actions-runner-controller#4736 · 2 commenti ·
I maintainer di solito rispondono entro 1 giorno
-
EphemeralRunnerSet creates more EphemeralRunners than `spec.replicas` on scale up (stale cache, no create expectations)Forse già presa @yaronr l’ha presa 29 giorni fa. Aperta
Difficoltà 4/5 3-5 giorni Idoneità per principianti 55/100
actions/actions-runner-controller#4722 · 1 commento ·
I maintainer di solito rispondono entro 1 giorno
Tutte le issue di actions/actions-runner-controller
Issue simili
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 77/100
mozilla-ai/cq#601 ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 84/100
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 78/100
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 78/100
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 1/5 1-3 ore Idoneità per principianti 78/100
I maintainer di solito rispondono entro 1 giorno