0.14.2: AutoscalingListener can retain deleted EphemeralRunnerSet after runner template update

Abierto
#4,561 9 comentarios 4 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
48/100
Tipo de issue
Error
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
github-actions, go, kubernetes

Línea de trabajo

Empieza rastreando la reconciliación de AutoscalingRunnerSet, AutoscalingListener y EphemeralRunnerSet durante --update-strategy=eventual, usando la actualización reproducida de runner-template y el gist de los logs del controlador. Se considera terminado cuando un listener ya no conserva el nombre de un EphemeralRunnerSet eliminado y el pod del listener se recupera sin eliminar manualmente el CR.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

stale
Checks
Controller Version

0.14.2

Deployment Method

ArgoCD

Checks
  • This isn't a question or user support case.
  • I've read the changelog and this is not an expected backward-incompatible change.
To Reproduce
  1. Deploy gha-runner-scale-set-controller and gha-runner-scale-set 0.14.2 with --update-strategy=eventual, minRunners: 1, and maxRunners: 10.
  2. Wait until the AutoscalingRunnerSet, AutoscalingListener, and EphemeralRunnerSet are healthy.
  3. Change AutoscalingRunnerSet.spec.template.spec so ARC rolls the EphemeralRunnerSet. In our case, a GitOps deployment removed an obsolete runtimeClassName and changed the runner container security context.
  4. Observe that the old EphemeralRunnerSet is deleted and a different EphemeralRunnerSet survives, while AutoscalingListener.spec.ephemeralRunnerSetName still names the deleted set.
  5. The controller repeatedly recreates the listener pod from that unchanged AutoscalingListener. Every pod attempts its initial patch against the deleted EphemeralRunnerSet, exits, and enters a permanent recreate/crash loop.

This happened during an update to an existing scale set, rather than a fresh install.

Describe the bug

On ARC 0.14.2, an AutoscalingListener remained bound to a deleted EphemeralRunnerSet after a runner-template update. The surviving EphemeralRunnerSet had a different generated name, but the listener CR, config, and RBAC still referenced the deleted name.

Every listener restart failed identically:

Application returned an error: handling initial message failed: could not patch ephemeral runner set,
patch JSON: {"spec":{"patchID":0,"replicas":1}},
error: ephemeralrunnersets.actions.github.com "<scale-set>-qb49n" not found

The controller then deleted and recreated the listener pod, but did not repair or replace the stale AutoscalingListener CR, so the loop continued indefinitely and jobs could not be assigned.

The symptoms and stale-reference mechanism match #4489 very closely. That issue covered 0.14.0/0.14.1 fresh installs and was closed by stale automation without a linked fix. This report confirms the same failure on 0.14.2 and during an update to an existing AutoscalingRunnerSet.

Recovery/workaround:

  1. Detect that AutoscalingListener.spec.ephemeralRunnerSetName is not the single active EphemeralRunnerSet.
  2. Delete the stale AutoscalingListener CR.
  3. With updateStrategy=eventual, ARC drains running jobs, creates the replacement EphemeralRunnerSet/listener, and recovers.

We currently run that check as a post-sync hook because the controller did not self-heal the stale reference.

Describe the expected behavior

After replacing an EphemeralRunnerSet, ARC should atomically create/update the listener against the surviving set, or reconcile and replace any listener whose spec.ephemeralRunnerSetName points to a missing/non-current set.

A listener pod restart should not repeatedly inherit a permanently stale generated resource name.

Additional Context
controllerVersion: 0.14.2
scaleSetChartVersion: 0.14.2
kubernetesVersion: v1.35.2
controllerArgs:
  - --auto-scaling-runner-set-only
  - --watch-single-namespace=<runner-namespace>
  - --runner-max-concurrent-reconciles=2
  - --update-strategy=eventual
scaleSet:
  minRunners: 1
  maxRunners: 10
observedState:
  listenerTarget: <scale-set>-qb49n
  listenerTargetExists: false
  activeEphemeralRunnerSet: <different generated name>
relatedIssues:
  - https://github.com/actions/actions-runner-controller/issues/4489
  - https://github.com/actions/actions-runner-controller/issues/4432
Controller Logs

Complete captured controller log window (the controller includes the terminated listener's full output in message):

https://gist.github.com/DocX/80b533f37c0bd1a5253fdd8514df6609

The first entry already contains the decisive listener error; the recreate loop continues throughout the capture.

Runner Pod Logs

No runner pod failure was involved. The failing pod was the AutoscalingListener, and its complete output is embedded in the controller termination messages in the gist above. Runner creation/assignment was blocked because the listener could not patch its referenced EphemeralRunnerSet.

Lenguaje dominante
Go
Estrellas
6.7k
Forks
1.5k
Merge medio
1 h 34 min
PR fusionados (30 d)
5

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de actions/actions-runner-controller

Todos los issues de actions/actions-runner-controller

Issues similares

Más issues de Go

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.