0.14.2: AutoscalingListener can retain deleted EphemeralRunnerSet after runner template update
Los mantenedores suelen responder en 1 día
Nadie ha tomado este issue todavía.
Evaluación
- Dificultad
- 4/5
- Tiempo estimado
- 3-5 días
- Aptitud para principiantes
- 48/100
- Tipo de issue
- Error
- Claridad
- Bastante claro
- Estado de actividad
- Activo
- Stack tecnológico
- github-actions, go, kubernetes
- Área
- devops, infrastructure
Línea de trabajo
Empieza rastreando la reconciliación de AutoscalingRunnerSet, AutoscalingListener y EphemeralRunnerSet durante --update-strategy=eventual, usando la actualización reproducida de runner-template y el gist de los logs del controlador. Se considera terminado cuando un listener ya no conserva el nombre de un EphemeralRunnerSet eliminado y el pod del listener se recupera sin eliminar manualmente el CR.
Escrito por el modelo de indexación a partir del texto del issue.
Descripción
Checks
- I've already read https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/troubleshooting-actions-runner-controller-errors and I'm sure my issue is not covered in the troubleshooting guide.
- I am using charts that are officially provided (rendered from the official 0.14.2 charts and applied through Argo CD).
Controller Version
0.14.2
Deployment Method
ArgoCD
Checks
- This isn't a question or user support case.
- I've read the changelog and this is not an expected backward-incompatible change.
To Reproduce
- Deploy
gha-runner-scale-set-controllerandgha-runner-scale-set0.14.2 with--update-strategy=eventual,minRunners: 1, andmaxRunners: 10. - Wait until the
AutoscalingRunnerSet,AutoscalingListener, andEphemeralRunnerSetare healthy. - Change
AutoscalingRunnerSet.spec.template.specso ARC rolls the EphemeralRunnerSet. In our case, a GitOps deployment removed an obsoleteruntimeClassNameand changed the runner container security context. - Observe that the old EphemeralRunnerSet is deleted and a different EphemeralRunnerSet survives, while
AutoscalingListener.spec.ephemeralRunnerSetNamestill names the deleted set. - The controller repeatedly recreates the listener pod from that unchanged
AutoscalingListener. Every pod attempts its initial patch against the deleted EphemeralRunnerSet, exits, and enters a permanent recreate/crash loop.
This happened during an update to an existing scale set, rather than a fresh install.
Describe the bug
On ARC 0.14.2, an AutoscalingListener remained bound to a deleted EphemeralRunnerSet after a runner-template update. The surviving EphemeralRunnerSet had a different generated name, but the listener CR, config, and RBAC still referenced the deleted name.
Every listener restart failed identically:
Application returned an error: handling initial message failed: could not patch ephemeral runner set,
patch JSON: {"spec":{"patchID":0,"replicas":1}},
error: ephemeralrunnersets.actions.github.com "<scale-set>-qb49n" not found
The controller then deleted and recreated the listener pod, but did not repair or replace the stale AutoscalingListener CR, so the loop continued indefinitely and jobs could not be assigned.
The symptoms and stale-reference mechanism match #4489 very closely. That issue covered 0.14.0/0.14.1 fresh installs and was closed by stale automation without a linked fix. This report confirms the same failure on 0.14.2 and during an update to an existing AutoscalingRunnerSet.
Recovery/workaround:
- Detect that
AutoscalingListener.spec.ephemeralRunnerSetNameis not the single active EphemeralRunnerSet. - Delete the stale AutoscalingListener CR.
- With
updateStrategy=eventual, ARC drains running jobs, creates the replacement EphemeralRunnerSet/listener, and recovers.
We currently run that check as a post-sync hook because the controller did not self-heal the stale reference.
Describe the expected behavior
After replacing an EphemeralRunnerSet, ARC should atomically create/update the listener against the surviving set, or reconcile and replace any listener whose spec.ephemeralRunnerSetName points to a missing/non-current set.
A listener pod restart should not repeatedly inherit a permanently stale generated resource name.
Additional Context
controllerVersion: 0.14.2
scaleSetChartVersion: 0.14.2
kubernetesVersion: v1.35.2
controllerArgs:
- --auto-scaling-runner-set-only
- --watch-single-namespace=<runner-namespace>
- --runner-max-concurrent-reconciles=2
- --update-strategy=eventual
scaleSet:
minRunners: 1
maxRunners: 10
observedState:
listenerTarget: <scale-set>-qb49n
listenerTargetExists: false
activeEphemeralRunnerSet: <different generated name>
relatedIssues:
- https://github.com/actions/actions-runner-controller/issues/4489
- https://github.com/actions/actions-runner-controller/issues/4432
Controller Logs
Complete captured controller log window (the controller includes the terminated listener's full output in message):
https://gist.github.com/DocX/80b533f37c0bd1a5253fdd8514df6609
The first entry already contains the decisive listener error; the recreate loop continues throughout the capture.
Runner Pod Logs
No runner pod failure was involved. The failing pod was the AutoscalingListener, and its complete output is embedded in the controller termination messages in the gist above. Runner creation/assignment was blocked because the listener could not patch its referenced EphemeralRunnerSet.
- Lenguaje dominante
- Go
- Estrellas
- 6.7k
- Forks
- 1.5k
- Merge medio
- 1 h 34 min
- PR fusionados (30 d)
- 5
Preparar el entorno
- Incluye un Dockerfile o un archivo de Docker Compose
- Sin plantilla de pull request
- Leer la guía de contribución
Primeros pasos
- Lee el issue completo y luego la guía de contribución del proyecto.
- Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
- Haz un fork del repositorio y trabaja en una rama.
- Abre un pull request que haga referencia al número del issue.
Más de actions/actions-runner-controller
-
community enhancement needs triage
Dificultad 2/5 1-3 horas Aptitud para principiantes 68/100
actions/actions-runner-controller#3997 · 9 comentarios · 27 reacciones ·
Los mantenedores suelen responder en 1 día
-
Surface the Outdated phase, and tell a runner version deprecation apart from a rejected specAbierto
Dificultad 4/5 3-5 días Aptitud para principiantes 48/100
actions/actions-runner-controller#4739 · 1 comentario · 4 reacciones ·
Los mantenedores suelen responder en 1 día
-
gha-runner-scale-set: support Google Cloud Secret Manager as a vault for GitHub credentialsPosiblemente ocupada Un pull request vinculado a esta issue está abierto o ya se fusionó. Abiertocommunity enhancement needs triage
Dificultad 4/5 3-5 días Aptitud para principiantes 52/100
actions/actions-runner-controller#4737 · 1 comentario ·
Los mantenedores suelen responder en 1 día
-
Controller upgrade cancels jobs that started shortly before their runner set is deletedQuizá libre de nuevo @ankit373 la tomó hace 9 días y no hay ningún pull request abierto. Abierto
Dificultad 4/5 3-5 días Aptitud para principiantes 45/100
actions/actions-runner-controller#4736 · 2 comentarios ·
Los mantenedores suelen responder en 1 día
-
EphemeralRunnerSet controller creates more runners than `spec.replicas` when its cache lags its own createsPosiblemente ocupada @jia-pylon la tomó hace 14 días. Abierto
Dificultad 4/5 3-5 días Aptitud para principiantes 30/100
actions/actions-runner-controller#4734 · 1 comentario ·
Los mantenedores suelen responder en 1 día
Todos los issues de actions/actions-runner-controller
Issues similares
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 82/100
-
needs discussion
Dificultad 2/5 1-3 horas Aptitud para principiantes 78/100
openfga/openfga.dev#1436 · 1 comentario ·
Los mantenedores suelen responder en 2 días
-
都道府県を指定すると区名から始まる住所に市名を補わないAbierto
Dificultad 2/5 1-3 horas Aptitud para principiantes 75/100
digital-go-jp/abr-geocoder#457 ·
Los mantenedores suelen responder en 3 días
-
benchmark
Dificultad 1/5 1-3 horas Aptitud para principiantes 72/100
Los mantenedores suelen responder en 1 día
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 77/100
mozilla-ai/cq#601 ·
Los mantenedores suelen responder en 1 día