0.14.2: AutoscalingListener can retain deleted EphemeralRunnerSet after runner template update
I maintainer di solito rispondono entro 1 giorno
Nessuno ha ancora preso questa issue.
Valutazione
- Difficoltà
- 4/5
- Tempo stimato
- 3-5 giorni
- Idoneità per principianti
- 48/100
- Tipo di issue
- Bug
- Chiarezza
- Abbastanza chiara
- Stato di attività
- Attiva
- Stack tecnologico
- github-actions, go, kubernetes
- Ambito
- devops, infrastructure
Direzione di ricerca
Inizia tracciando la riconciliazione di AutoscalingRunnerSet, AutoscalingListener ed EphemeralRunnerSet durante --update-strategy=eventual, usando l’aggiornamento riprodotto di runner-template e il gist dei log del controller. Il lavoro è completato quando un listener non conserva più il nome di un EphemeralRunnerSet eliminato e il pod del listener si ripristina senza eliminare manualmente il CR.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Descrizione
Checks
- I've already read https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/troubleshooting-actions-runner-controller-errors and I'm sure my issue is not covered in the troubleshooting guide.
- I am using charts that are officially provided (rendered from the official 0.14.2 charts and applied through Argo CD).
Controller Version
0.14.2
Deployment Method
ArgoCD
Checks
- This isn't a question or user support case.
- I've read the changelog and this is not an expected backward-incompatible change.
To Reproduce
- Deploy
gha-runner-scale-set-controllerandgha-runner-scale-set0.14.2 with--update-strategy=eventual,minRunners: 1, andmaxRunners: 10. - Wait until the
AutoscalingRunnerSet,AutoscalingListener, andEphemeralRunnerSetare healthy. - Change
AutoscalingRunnerSet.spec.template.specso ARC rolls the EphemeralRunnerSet. In our case, a GitOps deployment removed an obsoleteruntimeClassNameand changed the runner container security context. - Observe that the old EphemeralRunnerSet is deleted and a different EphemeralRunnerSet survives, while
AutoscalingListener.spec.ephemeralRunnerSetNamestill names the deleted set. - The controller repeatedly recreates the listener pod from that unchanged
AutoscalingListener. Every pod attempts its initial patch against the deleted EphemeralRunnerSet, exits, and enters a permanent recreate/crash loop.
This happened during an update to an existing scale set, rather than a fresh install.
Describe the bug
On ARC 0.14.2, an AutoscalingListener remained bound to a deleted EphemeralRunnerSet after a runner-template update. The surviving EphemeralRunnerSet had a different generated name, but the listener CR, config, and RBAC still referenced the deleted name.
Every listener restart failed identically:
Application returned an error: handling initial message failed: could not patch ephemeral runner set,
patch JSON: {"spec":{"patchID":0,"replicas":1}},
error: ephemeralrunnersets.actions.github.com "<scale-set>-qb49n" not found
The controller then deleted and recreated the listener pod, but did not repair or replace the stale AutoscalingListener CR, so the loop continued indefinitely and jobs could not be assigned.
The symptoms and stale-reference mechanism match #4489 very closely. That issue covered 0.14.0/0.14.1 fresh installs and was closed by stale automation without a linked fix. This report confirms the same failure on 0.14.2 and during an update to an existing AutoscalingRunnerSet.
Recovery/workaround:
- Detect that
AutoscalingListener.spec.ephemeralRunnerSetNameis not the single active EphemeralRunnerSet. - Delete the stale AutoscalingListener CR.
- With
updateStrategy=eventual, ARC drains running jobs, creates the replacement EphemeralRunnerSet/listener, and recovers.
We currently run that check as a post-sync hook because the controller did not self-heal the stale reference.
Describe the expected behavior
After replacing an EphemeralRunnerSet, ARC should atomically create/update the listener against the surviving set, or reconcile and replace any listener whose spec.ephemeralRunnerSetName points to a missing/non-current set.
A listener pod restart should not repeatedly inherit a permanently stale generated resource name.
Additional Context
controllerVersion: 0.14.2
scaleSetChartVersion: 0.14.2
kubernetesVersion: v1.35.2
controllerArgs:
- --auto-scaling-runner-set-only
- --watch-single-namespace=<runner-namespace>
- --runner-max-concurrent-reconciles=2
- --update-strategy=eventual
scaleSet:
minRunners: 1
maxRunners: 10
observedState:
listenerTarget: <scale-set>-qb49n
listenerTargetExists: false
activeEphemeralRunnerSet: <different generated name>
relatedIssues:
- https://github.com/actions/actions-runner-controller/issues/4489
- https://github.com/actions/actions-runner-controller/issues/4432
Controller Logs
Complete captured controller log window (the controller includes the terminated listener's full output in message):
https://gist.github.com/DocX/80b533f37c0bd1a5253fdd8514df6609
The first entry already contains the decisive listener error; the recreate loop continues throughout the capture.
Runner Pod Logs
No runner pod failure was involved. The failing pod was the AutoscalingListener, and its complete output is embedded in the controller termination messages in the gist above. Runner creation/assignment was blocked because the listener could not patch its referenced EphemeralRunnerSet.
- Lingua principale
- Go
- Stelle
- 6.7k
- Fork
- 1.5k
- Merge medio
- 1h 34m
- PR unite (30g)
- 5
Preparare l'ambiente
- Include un Dockerfile o un file Docker Compose
- Nessun modello di pull request
- Leggi la guida per i contributori
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Altre issue di actions/actions-runner-controller
-
community enhancement needs triage
Difficoltà 2/5 1-3 ore Idoneità per principianti 68/100
actions/actions-runner-controller#3997 · 9 commenti · 27 reazioni ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 4/5 3-5 giorni Idoneità per principianti 48/100
actions/actions-runner-controller#4739 · 1 commento · 4 reazioni ·
I maintainer di solito rispondono entro 1 giorno
-
gha-runner-scale-set: support Google Cloud Secret Manager as a vault for GitHub credentialsForse già presa Una pull request collegata a questa issue è aperta o già unita. Apertacommunity enhancement needs triage
Difficoltà 4/5 3-5 giorni Idoneità per principianti 52/100
actions/actions-runner-controller#4737 · 1 commento ·
I maintainer di solito rispondono entro 1 giorno
-
Controller upgrade cancels jobs that started shortly before their runner set is deletedForse di nuovo libera @ankit373 l’ha presa 9 giorni fa e non c’è nessuna pull request aperta. Aperta
Difficoltà 4/5 3-5 giorni Idoneità per principianti 45/100
actions/actions-runner-controller#4736 · 2 commenti ·
I maintainer di solito rispondono entro 1 giorno
-
EphemeralRunnerSet controller creates more runners than `spec.replicas` when its cache lags its own createsForse già presa @jia-pylon l’ha presa 14 giorni fa. Aperta
Difficoltà 4/5 3-5 giorni Idoneità per principianti 30/100
actions/actions-runner-controller#4734 · 1 commento ·
I maintainer di solito rispondono entro 1 giorno
Tutte le issue di actions/actions-runner-controller
Issue simili
-
benchmark
Difficoltà 1/5 1-3 ore Idoneità per principianti 72/100
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 77/100
mozilla-ai/cq#601 ·
I maintainer di solito rispondono entro 1 giorno
-
documentation migration
Difficoltà 2/5 1-3 ore Idoneità per principianti 75/100
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 84/100
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 78/100
I maintainer di solito rispondono entro 1 giorno