EphemeralRunnerSet creates more EphemeralRunners than `spec.replicas` on scale up (stale cache, no create expectations)

Open
#4,722 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

@yaronr is already working on this.

Since Oct 21, 2026.

  • #4723 by @yaronr — open
  • #4744 by @nikola-jokic — open

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
55/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
go, kubernetes

Research direction

Start with controllers/actions.github.com/ephemeralrunnerset_controller.go, especially the child listing and scale-up logic around the cited lines 233 and 356; then read the regression test described in the issue. Reproduce the partial-cache sequence with replicas set to 2 and a new patch ID, and verify the reconciler ends with exactly 2 runners. The issue offers two fix directions, so discuss the approach with maintainers before implementing.

Written by the indexing model from the issue text.

Description

Controller version

gha-runner-scale-set-controller 0.15.0 (commit 1945b42820760eb50893c8da77fdfc42cf8dad45). Our build adds a listener-side patch only; controllers/actions.github.com is unmodified.

What happened

The listener asked for 2 runners (Calculated target runner count "assigned job"=2 decision=2, replicas=2). The EphemeralRunnerSet controller created 3. Controller log, all within the same second:

Scaling comparison  current=0 desired=2
Creating new ephemeral runners (scale up)  count=2      -> runners A, B
Ephemeral runner counts  pending=1
Scaling comparison  current=1 desired=2
Creating new ephemeral runners (scale up)  count=1      -> runner C
Ephemeral runner counts  pending=3
Scaling comparison  current=3 desired=2
Deleting ephemeral runners (scale down)  count=1
Skipping ephemeral runner since its runner ID is not recorded yet ... retryAfter=59s   (x3)

Each extra runner pod requests real capacity, so the cluster autoscaler adds a node for it. The scale down cannot remove it until its runner ID is recorded. It is intermittent: an earlier run on the same day created exactly 2 for 2 jobs.

Analysis

  1. Reconcile counts child runners from the informer cache (ephemeralrunnerset_controller.go:233, r.List with MatchingFields{resourceOwnerKey}) and computes count := Spec.Replicas - scaleUpTotal (:356). Nothing records the runners this reconciler just created (no expectations as in the ReplicaSet controller), so a reconcile that runs before the cache holds all of them creates the missing ones again.
  2. The reconcile that sees the partial cache is the one enqueued by the watch event of the first created runner (Owns(&EphemeralRunner{}), :1203). It runs as soon as the creating reconcile returns. It reaches the scaling block only when Spec.PatchID is 0 or differs from the newest patch ID on the visible runners (:313), so the second ingredient is a listener patch (same replicas, new patchID) that lands inside that window. When the next patch comes later, the follow-up reconcile skips scaling and the count is exact: in a run that created exactly 2, the follow-up reconcile 1 s after the creates saw pending 2 and the listener re-patched (patchID 2, then 3) about 50 s later.
  3. This gap existed in 0.14.2, but 0.14.2 made it narrow:
    • creates were sequential (0.14.2 :380), and
    • updateStatus always issued a status PATCH after a create, because CurrentReplicas changed and was forced (CurrentReplicas = -1 // ALWAYS update current replicas, 0.14.2 :262). That round trip gave the watch time to deliver the last create before Reconcile returned.
  4. 0.15.0 removed both delays:
    • #4673 creates in parallel (:1000, errgroup, runnerBatchConcurrency = 8 at :68), so all creates return at nearly the same time;
    • #4557 removed the counts from the status, so updateStatus skips the PATCH when the phase is unchanged (:683).
      Reconcile now returns right after the last Create response, and the next reconcile lists before the watch has delivered the other runners.
  5. ephemeralRunnerSetMaxConcurrentReconciles does not help: controller-runtime never reconciles one key concurrently. The race is between sequential reconciles of the same set.

Regression test

A fake-client test reproduces it without timing: reconcile with replicas: 2, let the cache-backed client list only the first created runner, bump spec.patchID, reconcile again. Unpatched 0.15.0 ends with 3 runners; with the first fix below it ends with 2. We are happy to open a PR with the fix and the test.

Suggested fix

Either of:

  • Confirm a scale up with an uncached read before creating: when scaleUpTotal < Spec.Replicas, list the children through APIReader (the reconciler already has one; filter ownership in process as patchAppliedActionableRevisionStatus does, since resourceOwnerKey is a cache-only index) and recompute. Cost: one namespace LIST per scale-up decision, not per reconcile. We run this as a local patch.
  • Add create expectations keyed by the EphemeralRunnerSet (record N on create, lower on each observed add, skip scale up while expectations are pending, with a timeout), as the ReplicaSet controller does.

Environment

Kubernetes v1.35.9 (Kamaji hosted control plane), one EphemeralRunnerSet per scale set, scale from zero.

Dominant language
Go
Stars
6.7k
Forks
1.5k
Avg merge
1h 34m
Merged PRs (30d)
5

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from actions/actions-runner-controller

All issues in actions/actions-runner-controller

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.