EphemeralRunnerSet creates more EphemeralRunners than `spec.replicas` on scale up (stale cache, no create expectations)

Aperta
#4,722 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

@yaronr ci sta già lavorando.

Dal 21/10/2026.

  • #4723 di @yaronr — aperta
  • #4744 di @nikola-jokic — aperta

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
55/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
go, kubernetes

Direzione di ricerca

Start with controllers/actions.github.com/ephemeralrunnerset_controller.go, especially the child listing and scale-up logic around the cited lines 233 and 356; then read the regression test described in the issue. Reproduce the partial-cache sequence with replicas set to 2 and a new patch ID, and verify the reconciler ends with exactly 2 runners. The issue offers two fix directions, so discuss the approach with maintainers before implementing.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Controller version

gha-runner-scale-set-controller 0.15.0 (commit 1945b42820760eb50893c8da77fdfc42cf8dad45). Our build adds a listener-side patch only; controllers/actions.github.com is unmodified.

What happened

The listener asked for 2 runners (Calculated target runner count "assigned job"=2 decision=2, replicas=2). The EphemeralRunnerSet controller created 3. Controller log, all within the same second:

Scaling comparison  current=0 desired=2
Creating new ephemeral runners (scale up)  count=2      -> runners A, B
Ephemeral runner counts  pending=1
Scaling comparison  current=1 desired=2
Creating new ephemeral runners (scale up)  count=1      -> runner C
Ephemeral runner counts  pending=3
Scaling comparison  current=3 desired=2
Deleting ephemeral runners (scale down)  count=1
Skipping ephemeral runner since its runner ID is not recorded yet ... retryAfter=59s   (x3)

Each extra runner pod requests real capacity, so the cluster autoscaler adds a node for it. The scale down cannot remove it until its runner ID is recorded. It is intermittent: an earlier run on the same day created exactly 2 for 2 jobs.

Analysis

  1. Reconcile counts child runners from the informer cache (ephemeralrunnerset_controller.go:233, r.List with MatchingFields{resourceOwnerKey}) and computes count := Spec.Replicas - scaleUpTotal (:356). Nothing records the runners this reconciler just created (no expectations as in the ReplicaSet controller), so a reconcile that runs before the cache holds all of them creates the missing ones again.
  2. The reconcile that sees the partial cache is the one enqueued by the watch event of the first created runner (Owns(&EphemeralRunner{}), :1203). It runs as soon as the creating reconcile returns. It reaches the scaling block only when Spec.PatchID is 0 or differs from the newest patch ID on the visible runners (:313), so the second ingredient is a listener patch (same replicas, new patchID) that lands inside that window. When the next patch comes later, the follow-up reconcile skips scaling and the count is exact: in a run that created exactly 2, the follow-up reconcile 1 s after the creates saw pending 2 and the listener re-patched (patchID 2, then 3) about 50 s later.
  3. This gap existed in 0.14.2, but 0.14.2 made it narrow:
    • creates were sequential (0.14.2 :380), and
    • updateStatus always issued a status PATCH after a create, because CurrentReplicas changed and was forced (CurrentReplicas = -1 // ALWAYS update current replicas, 0.14.2 :262). That round trip gave the watch time to deliver the last create before Reconcile returned.
  4. 0.15.0 removed both delays:
    • #4673 creates in parallel (:1000, errgroup, runnerBatchConcurrency = 8 at :68), so all creates return at nearly the same time;
    • #4557 removed the counts from the status, so updateStatus skips the PATCH when the phase is unchanged (:683).
      Reconcile now returns right after the last Create response, and the next reconcile lists before the watch has delivered the other runners.
  5. ephemeralRunnerSetMaxConcurrentReconciles does not help: controller-runtime never reconciles one key concurrently. The race is between sequential reconciles of the same set.

Regression test

A fake-client test reproduces it without timing: reconcile with replicas: 2, let the cache-backed client list only the first created runner, bump spec.patchID, reconcile again. Unpatched 0.15.0 ends with 3 runners; with the first fix below it ends with 2. We are happy to open a PR with the fix and the test.

Suggested fix

Either of:

  • Confirm a scale up with an uncached read before creating: when scaleUpTotal < Spec.Replicas, list the children through APIReader (the reconciler already has one; filter ownership in process as patchAppliedActionableRevisionStatus does, since resourceOwnerKey is a cache-only index) and recompute. Cost: one namespace LIST per scale-up decision, not per reconcile. We run this as a local patch.
  • Add create expectations keyed by the EphemeralRunnerSet (record N on create, lower on each observed add, skip scale up while expectations are pending, with a timeout), as the ReplicaSet controller does.

Environment

Kubernetes v1.35.9 (Kamaji hosted control plane), one EphemeralRunnerSet per scale set, scale from zero.

Lingua principale
Go
Stelle
6.7k
Fork
1.5k
Merge medio
1h 34m
PR unite (30g)
5

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di actions/actions-runner-controller

Tutte le issue di actions/actions-runner-controller

Issue simili

Altre issue su Go

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.