EphemeralRunnerSet creates more EphemeralRunners than `spec.replicas` on scale up (stale cache, no create expectations)
Maintainers usually reply within 1 day
@yaronr is already working on this.
Since Oct 21, 2026.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 55/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Active
- Tech stack
- go, kubernetes
- Domain
- devops, infrastructure
Research direction
Start with controllers/actions.github.com/ephemeralrunnerset_controller.go, especially the child listing and scale-up logic around the cited lines 233 and 356; then read the regression test described in the issue. Reproduce the partial-cache sequence with replicas set to 2 and a new patch ID, and verify the reconciler ends with exactly 2 runners. The issue offers two fix directions, so discuss the approach with maintainers before implementing.
Written by the indexing model from the issue text.
Description
Controller version
gha-runner-scale-set-controller 0.15.0 (commit 1945b42820760eb50893c8da77fdfc42cf8dad45). Our build adds a listener-side patch only; controllers/actions.github.com is unmodified.
What happened
The listener asked for 2 runners (Calculated target runner count "assigned job"=2 decision=2, replicas=2). The EphemeralRunnerSet controller created 3. Controller log, all within the same second:
Scaling comparison current=0 desired=2
Creating new ephemeral runners (scale up) count=2 -> runners A, B
Ephemeral runner counts pending=1
Scaling comparison current=1 desired=2
Creating new ephemeral runners (scale up) count=1 -> runner C
Ephemeral runner counts pending=3
Scaling comparison current=3 desired=2
Deleting ephemeral runners (scale down) count=1
Skipping ephemeral runner since its runner ID is not recorded yet ... retryAfter=59s (x3)
Each extra runner pod requests real capacity, so the cluster autoscaler adds a node for it. The scale down cannot remove it until its runner ID is recorded. It is intermittent: an earlier run on the same day created exactly 2 for 2 jobs.
Analysis
Reconcilecounts child runners from the informer cache (ephemeralrunnerset_controller.go:233,r.ListwithMatchingFields{resourceOwnerKey}) and computescount := Spec.Replicas - scaleUpTotal(:356). Nothing records the runners this reconciler just created (no expectations as in the ReplicaSet controller), so a reconcile that runs before the cache holds all of them creates the missing ones again.- The reconcile that sees the partial cache is the one enqueued by the watch event of the first created runner (
Owns(&EphemeralRunner{}),:1203). It runs as soon as the creating reconcile returns. It reaches the scaling block only whenSpec.PatchIDis 0 or differs from the newest patch ID on the visible runners (:313), so the second ingredient is a listener patch (samereplicas, newpatchID) that lands inside that window. When the next patch comes later, the follow-up reconcile skips scaling and the count is exact: in a run that created exactly 2, the follow-up reconcile 1 s after the creates sawpending 2and the listener re-patched (patchID2, then 3) about 50 s later. - This gap existed in 0.14.2, but 0.14.2 made it narrow:
- creates were sequential (0.14.2
:380), and updateStatusalways issued a status PATCH after a create, becauseCurrentReplicaschanged and was forced (CurrentReplicas = -1 // ALWAYS update current replicas, 0.14.2:262). That round trip gave the watch time to deliver the last create beforeReconcilereturned.
- creates were sequential (0.14.2
- 0.15.0 removed both delays:
- #4673 creates in parallel (
:1000,errgroup,runnerBatchConcurrency = 8at:68), so all creates return at nearly the same time; - #4557 removed the counts from the status, so
updateStatusskips the PATCH when the phase is unchanged (:683).
Reconcilenow returns right after the lastCreateresponse, and the next reconcile lists before the watch has delivered the other runners.
- #4673 creates in parallel (
ephemeralRunnerSetMaxConcurrentReconcilesdoes not help: controller-runtime never reconciles one key concurrently. The race is between sequential reconciles of the same set.
Regression test
A fake-client test reproduces it without timing: reconcile with replicas: 2, let the cache-backed client list only the first created runner, bump spec.patchID, reconcile again. Unpatched 0.15.0 ends with 3 runners; with the first fix below it ends with 2. We are happy to open a PR with the fix and the test.
Suggested fix
Either of:
- Confirm a scale up with an uncached read before creating: when
scaleUpTotal < Spec.Replicas, list the children throughAPIReader(the reconciler already has one; filter ownership in process aspatchAppliedActionableRevisionStatusdoes, sinceresourceOwnerKeyis a cache-only index) and recompute. Cost: one namespace LIST per scale-up decision, not per reconcile. We run this as a local patch. - Add create expectations keyed by the EphemeralRunnerSet (record N on create, lower on each observed add, skip scale up while expectations are pending, with a timeout), as the ReplicaSet controller does.
Environment
Kubernetes v1.35.9 (Kamaji hosted control plane), one EphemeralRunnerSet per scale set, scale from zero.
- Dominant language
- Go
- Stars
- 6.7k
- Forks
- 1.5k
- Avg merge
- 1h 34m
- Merged PRs (30d)
- 5
Getting set up
- Ships a Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from actions/actions-runner-controller
-
community enhancement needs triage
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
actions/actions-runner-controller#3997 · 9 comments · 27 reactions ·
Maintainers usually reply within 1 day
-
Difficulty 4/5 3-5 days Newbie friendliness 48/100
actions/actions-runner-controller#4739 · 1 comment · 4 reactions ·
Maintainers usually reply within 1 day
-
gha-runner-scale-set: support Google Cloud Secret Manager as a vault for GitHub credentialsPossibly taken A pull request linked to this issue is open or already merged. Opencommunity enhancement needs triage
Difficulty 4/5 3-5 days Newbie friendliness 52/100
actions/actions-runner-controller#4737 · 1 comment ·
Maintainers usually reply within 1 day
-
Controller upgrade cancels jobs that started shortly before their runner set is deletedMay be free again @ankit373 claimed this 9 days ago, and no pull request is open. Open
Difficulty 4/5 3-5 days Newbie friendliness 45/100
actions/actions-runner-controller#4736 · 2 comments ·
Maintainers usually reply within 1 day
-
EphemeralRunnerSet controller creates more runners than `spec.replicas` when its cache lags its own createsPossibly taken @jia-pylon claimed this 14 days ago. Open
Difficulty 4/5 3-5 days Newbie friendliness 30/100
actions/actions-runner-controller#4734 · 1 comment ·
Maintainers usually reply within 1 day
All issues in actions/actions-runner-controller
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 77/100
mozilla-ai/cq#601 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day
-
Difficulty 1/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day