EphemeralRunnerSet creates more EphemeralRunners than `spec.replicas` on scale up (stale cache, no create expectations)

オープン
#4,722 コメント 1 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

@yaronr がすでに取り組んでいます。

2026年10月21日 から。

  • #4723 @yaronr による — オープン
  • #4744 @nikola-jokic による — オープン

評価

難易度
4/5
見積もり時間
3〜5日
初心者へのやさしさ
55/100
issue の種類
バグ
明瞭さ
明確に書かれている
活発さ
活発
技術スタック
go, kubernetes

調査の方向性

Start with controllers/actions.github.com/ephemeralrunnerset_controller.go, especially the child listing and scale-up logic around the cited lines 233 and 356; then read the regression test described in the issue. Reproduce the partial-cache sequence with replicas set to 2 and a new patch ID, and verify the reconciler ends with exactly 2 runners. The issue offers two fix directions, so discuss the approach with maintainers before implementing.

索引モデルが issue の本文から書いたものです。

説明

Controller version

gha-runner-scale-set-controller 0.15.0 (commit 1945b42820760eb50893c8da77fdfc42cf8dad45). Our build adds a listener-side patch only; controllers/actions.github.com is unmodified.

What happened

The listener asked for 2 runners (Calculated target runner count "assigned job"=2 decision=2, replicas=2). The EphemeralRunnerSet controller created 3. Controller log, all within the same second:

Scaling comparison  current=0 desired=2
Creating new ephemeral runners (scale up)  count=2      -> runners A, B
Ephemeral runner counts  pending=1
Scaling comparison  current=1 desired=2
Creating new ephemeral runners (scale up)  count=1      -> runner C
Ephemeral runner counts  pending=3
Scaling comparison  current=3 desired=2
Deleting ephemeral runners (scale down)  count=1
Skipping ephemeral runner since its runner ID is not recorded yet ... retryAfter=59s   (x3)

Each extra runner pod requests real capacity, so the cluster autoscaler adds a node for it. The scale down cannot remove it until its runner ID is recorded. It is intermittent: an earlier run on the same day created exactly 2 for 2 jobs.

Analysis

  1. Reconcile counts child runners from the informer cache (ephemeralrunnerset_controller.go:233, r.List with MatchingFields{resourceOwnerKey}) and computes count := Spec.Replicas - scaleUpTotal (:356). Nothing records the runners this reconciler just created (no expectations as in the ReplicaSet controller), so a reconcile that runs before the cache holds all of them creates the missing ones again.
  2. The reconcile that sees the partial cache is the one enqueued by the watch event of the first created runner (Owns(&EphemeralRunner{}), :1203). It runs as soon as the creating reconcile returns. It reaches the scaling block only when Spec.PatchID is 0 or differs from the newest patch ID on the visible runners (:313), so the second ingredient is a listener patch (same replicas, new patchID) that lands inside that window. When the next patch comes later, the follow-up reconcile skips scaling and the count is exact: in a run that created exactly 2, the follow-up reconcile 1 s after the creates saw pending 2 and the listener re-patched (patchID 2, then 3) about 50 s later.
  3. This gap existed in 0.14.2, but 0.14.2 made it narrow:
    • creates were sequential (0.14.2 :380), and
    • updateStatus always issued a status PATCH after a create, because CurrentReplicas changed and was forced (CurrentReplicas = -1 // ALWAYS update current replicas, 0.14.2 :262). That round trip gave the watch time to deliver the last create before Reconcile returned.
  4. 0.15.0 removed both delays:
    • #4673 creates in parallel (:1000, errgroup, runnerBatchConcurrency = 8 at :68), so all creates return at nearly the same time;
    • #4557 removed the counts from the status, so updateStatus skips the PATCH when the phase is unchanged (:683).
      Reconcile now returns right after the last Create response, and the next reconcile lists before the watch has delivered the other runners.
  5. ephemeralRunnerSetMaxConcurrentReconciles does not help: controller-runtime never reconciles one key concurrently. The race is between sequential reconciles of the same set.

Regression test

A fake-client test reproduces it without timing: reconcile with replicas: 2, let the cache-backed client list only the first created runner, bump spec.patchID, reconcile again. Unpatched 0.15.0 ends with 3 runners; with the first fix below it ends with 2. We are happy to open a PR with the fix and the test.

Suggested fix

Either of:

  • Confirm a scale up with an uncached read before creating: when scaleUpTotal < Spec.Replicas, list the children through APIReader (the reconciler already has one; filter ownership in process as patchAppliedActionableRevisionStatus does, since resourceOwnerKey is a cache-only index) and recompute. Cost: one namespace LIST per scale-up decision, not per reconcile. We run this as a local patch.
  • Add create expectations keyed by the EphemeralRunnerSet (record N on create, lower on each observed add, skip scale up while expectations are pending, with a timeout), as the ReplicaSet controller does.

Environment

Kubernetes v1.35.9 (Kamaji hosted control plane), one EphemeralRunnerSet per scale set, scale from zero.

主要言語
Go
スター
6.7k
フォーク
1.5k
平均マージ
1時間 34分
マージ済み PR(30日)
5

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

actions/actions-runner-controller のほかの issue

actions/actions-runner-controller の issue をすべて見る

似ている issue

Go の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。