EphemeralRunnerSet controller creates more runners than `spec.replicas` when its cache lags its own creates

Aperta
#4,734 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

@jia-pylon ci sta già lavorando.

Dal 4/11/2026.

  • #4735 di @jia-pylon — aperta
  • #4744 di @nikola-jokic — aperta

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
30/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Ferma
Stack tecnologico
go, kubernetes
Ambito
backend, devops

Direzione di ricerca

Start with EphemeralRunnerSetReconciler.Reconcile in the controller package and trace how it lists cached child runners before creating or deleting them. Run the envtest suite with the documented command and inspect the scale-up logs for counts that exceed spec.replicas. Done means reconciliation accounts for in-flight creates and deletes, with regression tests covering the cache lag; a linked pull request is already open.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Checks
Controller Version

gha-runner-scale-set-0.14.2; also reproduced on master 89f34c4f782d

Deployment Method

Helm

Checks
  • This isn't a question or user support case (For Q&A and community support, go to Discussions).
  • I've read the Changelog before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes
To Reproduce
  1. Check out master and run the controller envtest suite against any recent kube-apiserver (reproduced on 1.35.0, 1.36.2 and 1.37.0 from setup-envtest):
    KUBEBUILDER_ASSETS=... go test ./controllers/actions.github.com/ -count=1 -ginkgo.v
  2. Find a scale-up from 0 in the controller log, e.g. in It should create/add all required resources for a new EphemeralRunnerSet (finalizer).
Describe the bug

EphemeralRunnerSetReconciler.Reconcile lists child EphemeralRunners from the informer cache (MatchingFields{resourceOwnerKey: ...}) and creates desired - current runners. Nothing tracks creates or deletes that are still in flight. Its own status patch, or the add event for the first runner in a batch, can wake the next reconcile before the cache has seen the whole batch. That reconcile counts too few runners and creates another batch. It then deletes the surplus, so specs that wait for the final count still pass, but spec.replicas has been exceeded in the meantime. Scale-down has the same shape: a cache that hasn't caught up with deletes can lead to choosing more runners to delete.

Over-creates in unmodified full-suite runs on master (each run executes all 188 specs):

kube-apiserver full runs runs with an over-create in the log
1.35.0 8 6
1.36.2 8 8
1.37.0 8 7

The race sometimes surfaces as an assertion failure instead, e.g. Should replace finished ephemeral runners with new ones on 0.14.2: current: 0, desired: 2 → 2 created → status count 0 → current: 0, desired: 2 → 2 more created → 4 pending.

Describe the expected behavior

The controller never has more than spec.replicas runners because of its own stale reads, and never deletes more than the scale-down requires. kube-controller-manager's ReplicaSet controller solves the same problem with controller expectations.

Additional Context

In production, runner sets briefly over-provision runner pods. The extra runners are deleted again, but in the meantime they use node capacity.

Controller Logs

1.36.2 envtest, one namespace, all within 2026-11-04T20:23:49Z:

Scaling comparison: current=0, deleting=0, desired=5
Creating new ephemeral runners (scale up): count=5
Created new ephemeral runner  (x5, distinct names)
Scaling comparison: current=1, deleting=0, desired=5
Creating new ephemeral runners (scale up): count=4
Created new ephemeral runner  (x4, distinct names)
Scaling comparison: current=7, deleting=0, desired=5
Deleting ephemeral runners (scale down): count=2

Nine runners were created where five were desired. Full logs are available on request.

Runner Pod Logs

N/A (controller-side race)

I have a fix with regression tests ready and will open a PR that references this issue.

Lingua principale
Go
Stelle
6.7k
Fork
1.5k
Merge medio
1h 34m
PR unite (30g)
5

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di actions/actions-runner-controller

Tutte le issue di actions/actions-runner-controller

Issue simili

Altre issue su Go

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.