EphemeralRunnerSet controller creates more runners than `spec.replicas` when its cache lags its own creates

Open
#4,734 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

@jia-pylon is already working on this.

Since Nov 4, 2026.

  • #4735 by @jia-pylon — open
  • #4744 by @nikola-jokic — open

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
30/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Stale
Tech stack
go, kubernetes
Domain
backend, devops

Research direction

Start with EphemeralRunnerSetReconciler.Reconcile in the controller package and trace how it lists cached child runners before creating or deleting them. Run the envtest suite with the documented command and inspect the scale-up logs for counts that exceed spec.replicas. Done means reconciliation accounts for in-flight creates and deletes, with regression tests covering the cache lag; a linked pull request is already open.

Written by the indexing model from the issue text.

Description

Checks
Controller Version

gha-runner-scale-set-0.14.2; also reproduced on master 89f34c4f782d

Deployment Method

Helm

Checks
  • This isn't a question or user support case (For Q&A and community support, go to Discussions).
  • I've read the Changelog before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes
To Reproduce
  1. Check out master and run the controller envtest suite against any recent kube-apiserver (reproduced on 1.35.0, 1.36.2 and 1.37.0 from setup-envtest):
    KUBEBUILDER_ASSETS=... go test ./controllers/actions.github.com/ -count=1 -ginkgo.v
  2. Find a scale-up from 0 in the controller log, e.g. in It should create/add all required resources for a new EphemeralRunnerSet (finalizer).
Describe the bug

EphemeralRunnerSetReconciler.Reconcile lists child EphemeralRunners from the informer cache (MatchingFields{resourceOwnerKey: ...}) and creates desired - current runners. Nothing tracks creates or deletes that are still in flight. Its own status patch, or the add event for the first runner in a batch, can wake the next reconcile before the cache has seen the whole batch. That reconcile counts too few runners and creates another batch. It then deletes the surplus, so specs that wait for the final count still pass, but spec.replicas has been exceeded in the meantime. Scale-down has the same shape: a cache that hasn't caught up with deletes can lead to choosing more runners to delete.

Over-creates in unmodified full-suite runs on master (each run executes all 188 specs):

kube-apiserver full runs runs with an over-create in the log
1.35.0 8 6
1.36.2 8 8
1.37.0 8 7

The race sometimes surfaces as an assertion failure instead, e.g. Should replace finished ephemeral runners with new ones on 0.14.2: current: 0, desired: 2 → 2 created → status count 0 → current: 0, desired: 2 → 2 more created → 4 pending.

Describe the expected behavior

The controller never has more than spec.replicas runners because of its own stale reads, and never deletes more than the scale-down requires. kube-controller-manager's ReplicaSet controller solves the same problem with controller expectations.

Additional Context

In production, runner sets briefly over-provision runner pods. The extra runners are deleted again, but in the meantime they use node capacity.

Controller Logs

1.36.2 envtest, one namespace, all within 2026-11-04T20:23:49Z:

Scaling comparison: current=0, deleting=0, desired=5
Creating new ephemeral runners (scale up): count=5
Created new ephemeral runner  (x5, distinct names)
Scaling comparison: current=1, deleting=0, desired=5
Creating new ephemeral runners (scale up): count=4
Created new ephemeral runner  (x4, distinct names)
Scaling comparison: current=7, deleting=0, desired=5
Deleting ephemeral runners (scale down): count=2

Nine runners were created where five were desired. Full logs are available on request.

Runner Pod Logs

N/A (controller-side race)

I have a fix with regression tests ready and will open a PR that references this issue.

Dominant language
Go
Stars
6.7k
Forks
1.5k
Avg merge
1h 34m
Merged PRs (30d)
5

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from actions/actions-runner-controller

All issues in actions/actions-runner-controller

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.