EphemeralRunnerSet controller creates more runners than `spec.replicas` when its cache lags its own creates
Maintainers usually reply within 1 day
@jia-pylon is already working on this.
Since Nov 4, 2026.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 30/100
Research direction
Start with EphemeralRunnerSetReconciler.Reconcile in the controller package and trace how it lists cached child runners before creating or deleting them. Run the envtest suite with the documented command and inspect the scale-up logs for counts that exceed spec.replicas. Done means reconciliation accounts for in-flight creates and deletes, with regression tests covering the cache lag; a linked pull request is already open.
Written by the indexing model from the issue text.
Description
Checks
- I've already read https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/troubleshooting-actions-runner-controller-errors and I'm sure my issue is not covered in the troubleshooting guide.
- I am using charts that are officially provided
Controller Version
gha-runner-scale-set-0.14.2; also reproduced on master 89f34c4f782d
Deployment Method
Helm
Checks
- This isn't a question or user support case (For Q&A and community support, go to Discussions).
- I've read the Changelog before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes
To Reproduce
- Check out master and run the controller envtest suite against any recent kube-apiserver (reproduced on 1.35.0, 1.36.2 and 1.37.0 from
setup-envtest):
KUBEBUILDER_ASSETS=... go test ./controllers/actions.github.com/ -count=1 -ginkgo.v - Find a scale-up from 0 in the controller log, e.g. in
It should create/add all required resources for a new EphemeralRunnerSet (finalizer).
Describe the bug
EphemeralRunnerSetReconciler.Reconcile lists child EphemeralRunners from the informer cache (MatchingFields{resourceOwnerKey: ...}) and creates desired - current runners. Nothing tracks creates or deletes that are still in flight. Its own status patch, or the add event for the first runner in a batch, can wake the next reconcile before the cache has seen the whole batch. That reconcile counts too few runners and creates another batch. It then deletes the surplus, so specs that wait for the final count still pass, but spec.replicas has been exceeded in the meantime. Scale-down has the same shape: a cache that hasn't caught up with deletes can lead to choosing more runners to delete.
Over-creates in unmodified full-suite runs on master (each run executes all 188 specs):
| kube-apiserver | full runs | runs with an over-create in the log |
|---|---|---|
| 1.35.0 | 8 | 6 |
| 1.36.2 | 8 | 8 |
| 1.37.0 | 8 | 7 |
The race sometimes surfaces as an assertion failure instead, e.g. Should replace finished ephemeral runners with new ones on 0.14.2: current: 0, desired: 2 → 2 created → status count 0 → current: 0, desired: 2 → 2 more created → 4 pending.
Describe the expected behavior
The controller never has more than spec.replicas runners because of its own stale reads, and never deletes more than the scale-down requires. kube-controller-manager's ReplicaSet controller solves the same problem with controller expectations.
Additional Context
In production, runner sets briefly over-provision runner pods. The extra runners are deleted again, but in the meantime they use node capacity.
Controller Logs
1.36.2 envtest, one namespace, all within 2026-11-04T20:23:49Z:
Scaling comparison: current=0, deleting=0, desired=5
Creating new ephemeral runners (scale up): count=5
Created new ephemeral runner (x5, distinct names)
Scaling comparison: current=1, deleting=0, desired=5
Creating new ephemeral runners (scale up): count=4
Created new ephemeral runner (x4, distinct names)
Scaling comparison: current=7, deleting=0, desired=5
Deleting ephemeral runners (scale down): count=2
Nine runners were created where five were desired. Full logs are available on request.
Runner Pod Logs
N/A (controller-side race)
I have a fix with regression tests ready and will open a PR that references this issue.
- Dominant language
- Go
- Stars
- 6.7k
- Forks
- 1.5k
- Avg merge
- 1h 34m
- Merged PRs (30d)
- 5
Getting set up
- Ships a Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from actions/actions-runner-controller
-
community enhancement needs triage
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
actions/actions-runner-controller#3997 · 9 comments · 27 reactions ·
Maintainers usually reply within 1 day
-
Difficulty 4/5 3-5 days Newbie friendliness 48/100
actions/actions-runner-controller#4739 · 1 comment · 4 reactions ·
Maintainers usually reply within 1 day
-
gha-runner-scale-set: support Google Cloud Secret Manager as a vault for GitHub credentialsPossibly taken A pull request linked to this issue is open or already merged. Opencommunity enhancement needs triage
Difficulty 4/5 3-5 days Newbie friendliness 52/100
actions/actions-runner-controller#4737 · 1 comment ·
Maintainers usually reply within 1 day
-
Controller upgrade cancels jobs that started shortly before their runner set is deletedMay be free again @ankit373 claimed this 9 days ago, and no pull request is open. Open
Difficulty 4/5 3-5 days Newbie friendliness 45/100
actions/actions-runner-controller#4736 · 2 comments ·
Maintainers usually reply within 1 day
-
EphemeralRunnerSet creates more EphemeralRunners than `spec.replicas` on scale up (stale cache, no create expectations)Possibly taken @yaronr claimed this 29 days ago. Open
Difficulty 4/5 3-5 days Newbie friendliness 55/100
actions/actions-runner-controller#4722 · 1 comment ·
Maintainers usually reply within 1 day
All issues in actions/actions-runner-controller
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 77/100
mozilla-ai/cq#601 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day
-
Difficulty 1/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day