EphemeralRunnerSet creates more EphemeralRunners than `spec.replicas` on scale up (stale cache, no create expectations)
メンテナーはふだん 1 日以内に返信
評価
- 難易度
- 4/5
- 見積もり時間
- 3〜5日
- 初心者へのやさしさ
- 55/100
- issue の種類
- バグ
- 明瞭さ
- 明確に書かれている
- 活発さ
- 活発
- 技術スタック
- go, kubernetes
- 領域
- devops, infrastructure
調査の方向性
Start with controllers/actions.github.com/ephemeralrunnerset_controller.go, especially the child listing and scale-up logic around the cited lines 233 and 356; then read the regression test described in the issue. Reproduce the partial-cache sequence with replicas set to 2 and a new patch ID, and verify the reconciler ends with exactly 2 runners. The issue offers two fix directions, so discuss the approach with maintainers before implementing.
索引モデルが issue の本文から書いたものです。
説明
Controller version
gha-runner-scale-set-controller 0.15.0 (commit 1945b42820760eb50893c8da77fdfc42cf8dad45). Our build adds a listener-side patch only; controllers/actions.github.com is unmodified.
What happened
The listener asked for 2 runners (Calculated target runner count "assigned job"=2 decision=2, replicas=2). The EphemeralRunnerSet controller created 3. Controller log, all within the same second:
Scaling comparison current=0 desired=2
Creating new ephemeral runners (scale up) count=2 -> runners A, B
Ephemeral runner counts pending=1
Scaling comparison current=1 desired=2
Creating new ephemeral runners (scale up) count=1 -> runner C
Ephemeral runner counts pending=3
Scaling comparison current=3 desired=2
Deleting ephemeral runners (scale down) count=1
Skipping ephemeral runner since its runner ID is not recorded yet ... retryAfter=59s (x3)
Each extra runner pod requests real capacity, so the cluster autoscaler adds a node for it. The scale down cannot remove it until its runner ID is recorded. It is intermittent: an earlier run on the same day created exactly 2 for 2 jobs.
Analysis
Reconcilecounts child runners from the informer cache (ephemeralrunnerset_controller.go:233,r.ListwithMatchingFields{resourceOwnerKey}) and computescount := Spec.Replicas - scaleUpTotal(:356). Nothing records the runners this reconciler just created (no expectations as in the ReplicaSet controller), so a reconcile that runs before the cache holds all of them creates the missing ones again.- The reconcile that sees the partial cache is the one enqueued by the watch event of the first created runner (
Owns(&EphemeralRunner{}),:1203). It runs as soon as the creating reconcile returns. It reaches the scaling block only whenSpec.PatchIDis 0 or differs from the newest patch ID on the visible runners (:313), so the second ingredient is a listener patch (samereplicas, newpatchID) that lands inside that window. When the next patch comes later, the follow-up reconcile skips scaling and the count is exact: in a run that created exactly 2, the follow-up reconcile 1 s after the creates sawpending 2and the listener re-patched (patchID2, then 3) about 50 s later. - This gap existed in 0.14.2, but 0.14.2 made it narrow:
- creates were sequential (0.14.2
:380), and updateStatusalways issued a status PATCH after a create, becauseCurrentReplicaschanged and was forced (CurrentReplicas = -1 // ALWAYS update current replicas, 0.14.2:262). That round trip gave the watch time to deliver the last create beforeReconcilereturned.
- creates were sequential (0.14.2
- 0.15.0 removed both delays:
- #4673 creates in parallel (
:1000,errgroup,runnerBatchConcurrency = 8at:68), so all creates return at nearly the same time; - #4557 removed the counts from the status, so
updateStatusskips the PATCH when the phase is unchanged (:683).
Reconcilenow returns right after the lastCreateresponse, and the next reconcile lists before the watch has delivered the other runners.
- #4673 creates in parallel (
ephemeralRunnerSetMaxConcurrentReconcilesdoes not help: controller-runtime never reconciles one key concurrently. The race is between sequential reconciles of the same set.
Regression test
A fake-client test reproduces it without timing: reconcile with replicas: 2, let the cache-backed client list only the first created runner, bump spec.patchID, reconcile again. Unpatched 0.15.0 ends with 3 runners; with the first fix below it ends with 2. We are happy to open a PR with the fix and the test.
Suggested fix
Either of:
- Confirm a scale up with an uncached read before creating: when
scaleUpTotal < Spec.Replicas, list the children throughAPIReader(the reconciler already has one; filter ownership in process aspatchAppliedActionableRevisionStatusdoes, sinceresourceOwnerKeyis a cache-only index) and recompute. Cost: one namespace LIST per scale-up decision, not per reconcile. We run this as a local patch. - Add create expectations keyed by the EphemeralRunnerSet (record N on create, lower on each observed add, skip scale up while expectations are pending, with a timeout), as the ReplicaSet controller does.
Environment
Kubernetes v1.35.9 (Kamaji hosted control plane), one EphemeralRunnerSet per scale set, scale from zero.
- 主要言語
- Go
- スター
- 6.7k
- フォーク
- 1.5k
- 平均マージ
- 1時間 34分
- マージ済み PR(30日)
- 5
環境構築
- Dockerfile または Docker Compose ファイルあり
- プルリクエストのテンプレートなし
- コントリビューションガイドを読む
はじめの一歩
- issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
- 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
- リポジトリをフォークし、ブランチを切って変更します。
- issue 番号を参照したプルリクエストを送ります。
actions/actions-runner-controller のほかの issue
-
community enhancement needs triage
難易度 2/5 1〜3時間 初心者へのやさしさ 68/100
actions/actions-runner-controller#3997 · コメント 9 件 · リアクション 27 件 ·
メンテナーはふだん 1 日以内に返信
-
難易度 4/5 3〜5日 初心者へのやさしさ 48/100
actions/actions-runner-controller#4739 · コメント 1 件 · リアクション 4 件 ·
メンテナーはふだん 1 日以内に返信
-
gha-runner-scale-set: support Google Cloud Secret Manager as a vault for GitHub credentials対応中かも このイシューにリンクされたプルリクエストがオープン中、またはマージ済みです。 オープンcommunity enhancement needs triage
難易度 4/5 3〜5日 初心者へのやさしさ 52/100
actions/actions-runner-controller#4737 · コメント 1 件 ·
メンテナーはふだん 1 日以内に返信
-
Controller upgrade cancels jobs that started shortly before their runner set is deleted再び着手できるかも @ankit373 が 9 日前に担当しましたが、オープン中のプルリクエストはありません。 オープン
難易度 4/5 3〜5日 初心者へのやさしさ 45/100
actions/actions-runner-controller#4736 · コメント 2 件 ·
メンテナーはふだん 1 日以内に返信
-
EphemeralRunnerSet controller creates more runners than `spec.replicas` when its cache lags its own creates対応中かも @jia-pylon が 14 日前に担当しました。 オープン
難易度 4/5 3〜5日 初心者へのやさしさ 30/100
actions/actions-runner-controller#4734 · コメント 1 件 ·
メンテナーはふだん 1 日以内に返信
actions/actions-runner-controller の issue をすべて見る
似ている issue
-
難易度 2/5 1〜3時間 初心者へのやさしさ 82/100
-
needs discussion
難易度 2/5 1〜3時間 初心者へのやさしさ 78/100
openfga/openfga.dev#1436 · コメント 1 件 ·
メンテナーはふだん 2 日以内に返信
-
難易度 2/5 1〜3時間 初心者へのやさしさ 75/100
digital-go-jp/abr-geocoder#457 ·
メンテナーはふだん 3 日以内に返信
-
benchmark
難易度 1/5 1〜3時間 初心者へのやさしさ 72/100
メンテナーはふだん 1 日以内に返信
-
難易度 2/5 1〜3時間 初心者へのやさしさ 77/100
mozilla-ai/cq#601 ·
メンテナーはふだん 1 日以内に返信