EphemeralRunnerSet creates more EphemeralRunners than `spec.replicas` on scale up (stale cache, no create expectations)

Đang mở
#4,722 1 bình luận 0 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

@yaronr đang làm issue này rồi.

Từ ngày 21/10/2026.

  • #4723 của @yaronr — đang mở
  • #4744 của @nikola-jokic — đang mở

Đánh giá

Độ khó
4/5
Thời gian dự kiến
3-5 ngày
Mức phù hợp với người mới
55/100
Loại issue
Lỗi
Độ rõ ràng
Đặc tả rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
go, kubernetes
Lĩnh vực
devops, infrastructure

Hướng nghiên cứu

Start with controllers/actions.github.com/ephemeralrunnerset_controller.go, especially the child listing and scale-up logic around the cited lines 233 and 356; then read the regression test described in the issue. Reproduce the partial-cache sequence with replicas set to 2 and a new patch ID, and verify the reconciler ends with exactly 2 runners. The issue offers two fix directions, so discuss the approach with maintainers before implementing.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

Controller version

gha-runner-scale-set-controller 0.15.0 (commit 1945b42820760eb50893c8da77fdfc42cf8dad45). Our build adds a listener-side patch only; controllers/actions.github.com is unmodified.

What happened

The listener asked for 2 runners (Calculated target runner count "assigned job"=2 decision=2, replicas=2). The EphemeralRunnerSet controller created 3. Controller log, all within the same second:

Scaling comparison  current=0 desired=2
Creating new ephemeral runners (scale up)  count=2      -> runners A, B
Ephemeral runner counts  pending=1
Scaling comparison  current=1 desired=2
Creating new ephemeral runners (scale up)  count=1      -> runner C
Ephemeral runner counts  pending=3
Scaling comparison  current=3 desired=2
Deleting ephemeral runners (scale down)  count=1
Skipping ephemeral runner since its runner ID is not recorded yet ... retryAfter=59s   (x3)

Each extra runner pod requests real capacity, so the cluster autoscaler adds a node for it. The scale down cannot remove it until its runner ID is recorded. It is intermittent: an earlier run on the same day created exactly 2 for 2 jobs.

Analysis

  1. Reconcile counts child runners from the informer cache (ephemeralrunnerset_controller.go:233, r.List with MatchingFields{resourceOwnerKey}) and computes count := Spec.Replicas - scaleUpTotal (:356). Nothing records the runners this reconciler just created (no expectations as in the ReplicaSet controller), so a reconcile that runs before the cache holds all of them creates the missing ones again.
  2. The reconcile that sees the partial cache is the one enqueued by the watch event of the first created runner (Owns(&EphemeralRunner{}), :1203). It runs as soon as the creating reconcile returns. It reaches the scaling block only when Spec.PatchID is 0 or differs from the newest patch ID on the visible runners (:313), so the second ingredient is a listener patch (same replicas, new patchID) that lands inside that window. When the next patch comes later, the follow-up reconcile skips scaling and the count is exact: in a run that created exactly 2, the follow-up reconcile 1 s after the creates saw pending 2 and the listener re-patched (patchID 2, then 3) about 50 s later.
  3. This gap existed in 0.14.2, but 0.14.2 made it narrow:
    • creates were sequential (0.14.2 :380), and
    • updateStatus always issued a status PATCH after a create, because CurrentReplicas changed and was forced (CurrentReplicas = -1 // ALWAYS update current replicas, 0.14.2 :262). That round trip gave the watch time to deliver the last create before Reconcile returned.
  4. 0.15.0 removed both delays:
    • #4673 creates in parallel (:1000, errgroup, runnerBatchConcurrency = 8 at :68), so all creates return at nearly the same time;
    • #4557 removed the counts from the status, so updateStatus skips the PATCH when the phase is unchanged (:683).
      Reconcile now returns right after the last Create response, and the next reconcile lists before the watch has delivered the other runners.
  5. ephemeralRunnerSetMaxConcurrentReconciles does not help: controller-runtime never reconciles one key concurrently. The race is between sequential reconciles of the same set.

Regression test

A fake-client test reproduces it without timing: reconcile with replicas: 2, let the cache-backed client list only the first created runner, bump spec.patchID, reconcile again. Unpatched 0.15.0 ends with 3 runners; with the first fix below it ends with 2. We are happy to open a PR with the fix and the test.

Suggested fix

Either of:

  • Confirm a scale up with an uncached read before creating: when scaleUpTotal < Spec.Replicas, list the children through APIReader (the reconciler already has one; filter ownership in process as patchAppliedActionableRevisionStatus does, since resourceOwnerKey is a cache-only index) and recompute. Cost: one namespace LIST per scale-up decision, not per reconcile. We run this as a local patch.
  • Add create expectations keyed by the EphemeralRunnerSet (record N on create, lower on each observed add, skip scale up while expectations are pending, with a timeout), as the ReplicaSet controller does.

Environment

Kubernetes v1.35.9 (Kamaji hosted control plane), one EphemeralRunnerSet per scale set, scale from zero.

Ngôn ngữ chính
Go
Star
6.7k
Fork
1.5k
Merge trung bình
1 giờ 34 phút
Pull request đã merge (30 ngày)
5

Chuẩn bị môi trường

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của actions/actions-runner-controller

Tất cả issue của actions/actions-runner-controller

Issue tương tự

Thêm issue về Go

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.