Controller upgrade cancels jobs that started shortly before their runner set is deleted
メンテナーはふだん 1 日以内に返信
@ankit373 がすでに取り組んでいます。
2026年11月10日 から。
評価
- 難易度
- 4/5
- 見積もり時間
- 3〜5日
- 初心者へのやさしさ
- 45/100
調査の方向性
Start by tracing the scale set cleanup path that calls RemoveRunner and checks HasJob(), then review the ephemeral runner status update path that records status.jobId. Reproduce the timing window described during a controller upgrade and inspect the cleanup tests for busy runners. Done means a runner executing a job is preserved during scale set deletion, with a regression test covering a registered runner whose job ID has not yet been recorded.
索引モデルが issue の本文から書いたものです。
説明
Checks
- I've already read the troubleshooting guide and I'm sure my issue is not covered in it.
- I am using charts that are officially provided
Controller Version
0.15.1 (upgraded from 0.14.2)
Deployment Method
Kustomize (manifests rendered from the official charts), applied with kubectl apply --server-side; ArgoCD with selfHeal on stage and prod.
Checks
- This isn't a question or user support case.
- I've read the Changelog before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes
To Reproduce
- Run 0.14.2 with a scale set (
minRunners: 0,maxRunners: 20). - Dispatch jobs onto the scale set every 10 seconds.
- While jobs keep starting, upgrade the controller (CRDs and controller Deployment) to 0.15.1. The new controller deletes the 0.14.2 scale sets, since their
major.minordiffers. - Check the conclusion of the jobs that started in the seconds before their runner was deleted.
Describe the bug
A job that starts on a runner whose EphemeralRunner has no status.jobId yet is cancelled when the scale set is deleted. The controller does not see the job (HasJob() is false), calls RemoveRunner, the service accepts the removal although the runner is executing the job, and the pod is deleted. The job fails with Process completed with exit code 130. and The runner has received a shutdown signal.
Runners whose status.jobId was already recorded are skipped (Skipping ephemeral runner since it is running a job) and their jobs finish.
Describe the expected behavior
A runner executing a job is not removed while its scale set is deleted. Either RemoveRunner refuses a busy runner (JobStillRunningError), as the controller expects, or the controller treats a registered runner without a recorded job as possibly busy until the service confirms otherwise.
Additional Context
Related: #4668 states that a late or lost job patch only costs a wasted API call, since RemoveRunner is rejected with JobStillRunningError while the job runs; here the service accepted it. #4683 protects runners that have no runner ID yet; the cancelled runners had one. #3298 reported the same symptom on 0.8.2 without an upgrade.
Measured on a dev cluster (EKS) with a 0.14.2 to 0.15.1 upgrade, 300 and 900 second jobs on one scale set, three runs:
| Run | Jobs running when their scale set was deleted | With status.jobId at that point |
Cancelled | Job age when its runner was removed |
|---|---|---|---|---|
| 1 | 4 | 4 | 0 | n/a |
| 2 | 1 | 0 | 1 | 54s |
| 3 | 13 | 12 | 1 | 9s |
In run 2 the controller was restarted 21 seconds after the job started; in run 3 the job started 23 seconds after the new controller was applied, while the old listeners were being replaced. In both cases the runner had a runnerId but no jobId when the controller cleaned up the scale set.
Run 3, controller log for the cancelled runner (names shortened):
14:43:46Z INFO EphemeralRunner Updated ephemeral runner status
14:43:56Z INFO EphemeralRunnerSet Cleaning up the ephemeral runner
14:43:57Z INFO EphemeralRunnerSet Deleting ephemeral runner after removing from the service {"runnerId": ...}
14:43:57Z INFO EphemeralRunnerSet Deleted ephemeral runner
The job (started_at 14:43:48Z) failed at 14:43:58Z with exit code 130. In the same reconcile, the 12 runners with a recorded jobId were skipped and their jobs succeeded, 8 of them started during the swap as well.
Controller Logs
Run 3: controller log for the cancelled runner (8pjml), a runner that was skipped (77qj7) and their EphemeralRunnerSet
2026-11-05T14:43:46Z INFO EphemeralRunner Runner container is still running; updating ephemeral runner status {"version": "0.15.1", "ephemeralrunner": {"name":"infra-aws-core-2-default-rts7j-runner-77qj7","namespace":"infra"}}
2026-11-05T14:43:46Z INFO EphemeralRunner Runner container is still running; updating ephemeral runner status {"version": "0.15.1", "ephemeralrunner": {"name":"infra-aws-core-2-default-rts7j-runner-8pjml","namespace":"infra"}}
2026-11-05T14:43:46Z INFO EphemeralRunner Updated ephemeral runner status {"version": "0.15.1", "ephemeralrunner": {"name":"infra-aws-core-2-default-rts7j-runner-8pjml","namespace":"infra"}}
2026-11-05T14:43:46Z INFO EphemeralRunner Updating ephemeral runner status {"version": "0.15.1", "ephemeralrunner": {"name":"infra-aws-core-2-default-rts7j-runner-8pjml","namespace":"infra"}, "statusPhase": "Running", "statusReason": "", "statusMessage": "", "ready": true}
2026-11-05T14:43:56Z INFO EphemeralRunnerSet Clean up runner counts {"version": "0.15.1", "ephemeralrunnerset": {"name":"infra-aws-core-2-default-rts7j","namespace":"infra"}, "pending": 1, "running": 12, "finished": 0, "failed": 0, "deleting": 0, "outdated": 0}
2026-11-05T14:43:56Z INFO EphemeralRunnerSet Cleaning up the ephemeral runner {"version": "0.15.1", "ephemeralrunnerset": {"name":"infra-aws-core-2-default-rts7j","namespace":"infra"}, "name": "infra-aws-core-2-default-rts7j-runner-8pjml"}
2026-11-05T14:43:56Z INFO EphemeralRunnerSet Cleanup pending or running ephemeral runners {"version": "0.15.1", "ephemeralrunnerset": {"name":"infra-aws-core-2-default-rts7j","namespace":"infra"}}
2026-11-05T14:43:56Z INFO EphemeralRunnerSet Deleting resources {"version": "0.15.1", "ephemeralrunnerset": {"name":"infra-aws-core-2-default-rts7j","namespace":"infra"}}
2026-11-05T14:43:57Z ERROR EphemeralRunner Failed to update with finalizer set {"version": "0.15.1", "ephemeralrunner": {"name":"infra-aws-core-2-default-rts7j-runner-8pjml","namespace":"infra"}, "error": "EphemeralRunner.actions.github.com \"infra-aws-core-2-default-rts7j-runner-8pjml\" is invalid: metadata.finalizers: Forbidden: no new finalizers can be added if the object is being deleted, found new finalizers []string{\"ephemeralrunner.actions.github.com/runner-registration-finalizer\"}"}
2026-11-05T14:43:57Z ERROR Reconciler error {"controller": "ephemeralrunner", "controllerGroup": "actions.github.com", "controllerKind": "EphemeralRunner", "EphemeralRunner": {"name":"infra-aws-core-2-default-rts7j-runner-8pjml","namespace":"infra"}, "namespace": "infra", "name": "infra-aws-core-2-default-rts7j-runner-8pjml", "reconcileID": "fc326ee9-1604-4f13-bdd9-e02d853ac08b", "error": "EphemeralRunner.actions.github.com \"infra-aws-core-2-default-rts7j-runner-8pjml\" is invalid: metadata.finalizers: Forbidden: no new finalizers can be added if the object is being deleted, found new finalizers []string{\"ephemeralrunner.actions.github.com/runner-registration-finalizer\"}"}
2026-11-05T14:43:57Z INFO EphemeralRunner Adding finalizers {"version": "0.15.1", "ephemeralrunner": {"name":"infra-aws-core-2-default-rts7j-runner-8pjml","namespace":"infra"}}
2026-11-05T14:43:57Z INFO EphemeralRunner Cleaning up the runner jitconfig secret {"version": "0.15.1", "ephemeralrunner": {"name":"infra-aws-core-2-default-rts7j-runner-8pjml","namespace":"infra"}}
2026-11-05T14:43:57Z INFO EphemeralRunner Cleaning up the runner pod {"version": "0.15.1", "ephemeralrunner": {"name":"infra-aws-core-2-default-rts7j-runner-8pjml","namespace":"infra"}}
2026-11-05T14:43:57Z INFO EphemeralRunner Deleted jitconfig secret {"version": "0.15.1", "ephemeralrunner": {"name":"infra-aws-core-2-default-rts7j-runner-8pjml","namespace":"infra"}}
2026-11-05T14:43:57Z INFO EphemeralRunner Deleted the runner pod {"version": "0.15.1", "ephemeralrunner": {"name":"infra-aws-core-2-default-rts7j-runner-8pjml","namespace":"infra"}}
2026-11-05T14:43:57Z INFO EphemeralRunner Deleting the jitconfig secret {"version": "0.15.1", "ephemeralrunner": {"name":"infra-aws-core-2-default-rts7j-runner-8pjml","namespace":"infra"}}
2026-11-05T14:43:57Z INFO EphemeralRunner Deleting the runner pod {"version": "0.15.1", "ephemeralrunner": {"name":"infra-aws-core-2-default-rts7j-runner-8pjml","namespace":"infra"}}
2026-11-05T14:43:57Z INFO EphemeralRunner Finalizing ephemeral runner {"version": "0.15.1", "ephemeralrunner": {"name":"infra-aws-core-2-default-rts7j-runner-8pjml","namespace":"infra"}}
2026-11-05T14:43:57Z INFO EphemeralRunner Removed finalizer from ephemeral runner {"version": "0.15.1", "ephemeralrunner": {"name":"infra-aws-core-2-default-rts7j-runner-8pjml","namespace":"infra"}}
2026-11-05T14:43:57Z INFO EphemeralRunner Removing finalizer {"version": "0.15.1", "ephemeralrunner": {"name":"infra-aws-core-2-default-rts7j-runner-8pjml","namespace":"infra"}}
2026-11-05T14:43:57Z INFO EphemeralRunnerSet Deleted ephemeral runner {"version": "0.15.1", "ephemeralrunnerset": {"name":"infra-aws-core-2-default-rts7j","namespace":"infra"}, "name": "infra-aws-core-2-default-rts7j-runner-8pjml", "runnerId": 10721544}
2026-11-05T14:43:57Z INFO EphemeralRunnerSet Deleting ephemeral runner after removing from the service {"version": "0.15.1", "ephemeralrunnerset": {"name":"infra-aws-core-2-default-rts7j","namespace":"infra"}, "name": "infra-aws-core-2-default-rts7j-runner-8pjml", "runnerId": 10721544}
2026-11-05T14:43:57Z INFO EphemeralRunnerSet Deleting resources {"version": "0.15.1", "ephemeralrunnerset": {"name":"infra-aws-core-2-default-rts7j","namespace":"infra"}}
2026-11-05T14:43:57Z INFO EphemeralRunnerSet Skipping ephemeral runner since it is running a job {"version": "0.15.1", "ephemeralrunnerset": {"name":"infra-aws-core-2-default-rts7j","namespace":"infra"}, "name": "infra-aws-core-2-default-rts7j-runner-77qj7", "workflowRunId": 42047428183, "jobId": "3fa9d8e3-1d97-5989-9d6e-e4038410b5ba"}
2026-11-05T14:43:57Z INFO EphemeralRunnerSet Skipping ephemeral runner since it is running a job {"version": "0.15.1", "ephemeralrunnerset": {"name":"infra-aws-core-2-default-rts7j","namespace":"infra"}, "name": "infra-aws-core-2-default-rts7j-runner-77qj7", "workflowRunId": 42047428183, "jobId": "3fa9d8e3-1d97-5989-9d6e-e4038410b5ba"}
2026-11-05T14:43:57Z INFO EphemeralRunnerSet Skipping ephemeral runner since it is running a job {"version": "0.15.1", "ephemeralrunnerset": {"name":"infra-aws-core-2-default-rts7j","namespace":"infra"}, "name": "infra-aws-core-2-default-rts7j-runner-77qj7", "workflowRunId": 42047428183, "jobId": "3fa9d8e3-1d97-5989-9d6e-e4038410b5ba"}
2026-11-05T14:43:57Z INFO EphemeralRunnerSet Skipping ephemeral runner since it is running a job {"version": "0.15.1", "ephemeralrunnerset": {"name":"infra-aws-core-2-default-rts7j","namespace":"infra"}, "name": "infra-aws-core-2-default-rts7j-runner-77qj7", "workflowRunId": 42047428183, "jobId": "3fa9d8e3-1d97-5989-9d6e-e4038410b5ba"}
Runner Pod Logs
Run 3: job log of the cancelled job
2026-11-05T14:43:49.9103757Z Current runner version: '2.338.1'
2026-11-05T14:43:49.9114558Z Runner name: 'infra-aws-core-2-default-rts7j-runner-8pjml'
2026-11-05T14:43:49.9116421Z Runner group name: 'infra-dev-self-hosted-runners'
2026-11-05T14:43:49.9118353Z Machine name: 'infra-aws-core-2-default-rts7j-runner-8pjml'
2026-11-05T14:43:49.9124877Z ##[group]GITHUB_TOKEN Permissions
2026-11-05T14:43:49.9128855Z Metadata: read
2026-11-05T14:43:49.9129771Z ##[endgroup]
2026-11-05T14:43:49.9133335Z Secret source: Actions
2026-11-05T14:43:49.9134775Z Cache mode: write
2026-11-05T14:43:49.9136019Z Prepare workflow directory
2026-11-05T14:43:49.9691728Z Prepare all required actions
2026-11-05T14:43:49.9867996Z Complete job name: test (dev-aws-core-2-default)
2026-11-05T14:43:50.1409078Z A job started hook has been configured by the self-hosted runner administrator
2026-11-05T14:43:50.1669201Z ##[group]Run '/home/runner/hooks/label-runner-pod.sh'
2026-11-05T14:43:50.1689061Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2026-11-05T14:43:50.1691119Z ##[endgroup]
2026-11-05T14:43:50.6453779Z Client Version: v1.35.8
2026-11-05T14:43:50.6455320Z Kustomize Version: v5.7.1
2026-11-05T14:43:50.6456952Z Server Version: v1.36.4-eks-cfb47f5
2026-11-05T14:43:50.6459249Z Adding label 'camunda.cloud/workflow=Test-dev-self-hosted-runners' to the runner
2026-11-05T14:43:50.6647463Z Adding label 'camunda.cloud/workflow-repository=camunda-infra-core' to the runner
2026-11-05T14:43:50.8239381Z Adding label 'camunda.cloud/job-id=test' to the runner
2026-11-05T14:43:50.9302870Z Adding label 'camunda.cloud/run-id=42047639016' to the runner
2026-11-05T14:43:51.0760490Z Adding label 'camunda.cloud/run-attempt=1' to the runner
2026-11-05T14:43:51.1591572Z Adding label 'camunda.cloud/github-ref=heads-stage' to the runner
2026-11-05T14:43:51.3887508Z Adding label 'camunda.cloud/base-ref=not-applicable' to the runner
2026-11-05T14:43:51.5314290Z ##[group]Run # The runner name tells which fleet picked the job up (maint-* for the maintenance sets).
2026-11-05T14:43:51.5317635Z # The runner name tells which fleet picked the job up (maint-* for the maintenance sets).
2026-11-05T14:43:51.5320322Z echo "Running on ${RUNNER_NAME} (label ${LABEL}) at $(date -u +%FT%TZ)"
2026-11-05T14:43:51.5323417Z sleep "$SLEEP_SECONDS"
2026-11-05T14:43:51.5324700Z echo "Finished at $(date -u +%FT%TZ)"
2026-11-05T14:43:51.5339535Z shell: /usr/bin/bash -e {0}
2026-11-05T14:43:51.5340790Z env:
2026-11-05T14:43:51.5341730Z SLEEP_SECONDS: 900
2026-11-05T14:43:51.5342831Z LABEL: dev-aws-core-2-default
2026-11-05T14:43:51.5343908Z ##[endgroup]
2026-11-05T14:43:51.5419819Z Running on infra-aws-core-2-default-rts7j-runner-8pjml (label dev-aws-core-2-default) at 2026-11-05T14:43:51Z
2026-11-05T14:43:57.1238858Z ##[error]Process completed with exit code 130.
2026-11-05T14:43:57.1424296Z Cleaning up orphan processes
2026-11-05T14:43:57.1555198Z ##[error]The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.
- 主要言語
- Go
- スター
- 6.7k
- フォーク
- 1.5k
- 平均マージ
- 1時間 34分
- マージ済み PR(30日)
- 5
環境構築
- Dockerfile または Docker Compose ファイルあり
- プルリクエストのテンプレートなし
- コントリビューションガイドを読む
はじめの一歩
- issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
- 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
- リポジトリをフォークし、ブランチを切って変更します。
- issue 番号を参照したプルリクエストを送ります。
actions/actions-runner-controller のほかの issue
-
community enhancement needs triage
難易度 2/5 1〜3時間 初心者へのやさしさ 68/100
actions/actions-runner-controller#3997 · コメント 9 件 · リアクション 27 件 ·
メンテナーはふだん 1 日以内に返信
-
難易度 4/5 3〜5日 初心者へのやさしさ 48/100
actions/actions-runner-controller#4739 · コメント 1 件 · リアクション 4 件 ·
メンテナーはふだん 1 日以内に返信
-
gha-runner-scale-set: support Google Cloud Secret Manager as a vault for GitHub credentials対応中かも このイシューにリンクされたプルリクエストがオープン中、またはマージ済みです。 オープンcommunity enhancement needs triage
難易度 4/5 3〜5日 初心者へのやさしさ 52/100
actions/actions-runner-controller#4737 · コメント 1 件 ·
メンテナーはふだん 1 日以内に返信
-
EphemeralRunnerSet controller creates more runners than `spec.replicas` when its cache lags its own creates対応中かも @jia-pylon が 14 日前に担当しました。 オープン
難易度 4/5 3〜5日 初心者へのやさしさ 30/100
actions/actions-runner-controller#4734 · コメント 1 件 ·
メンテナーはふだん 1 日以内に返信
-
EphemeralRunnerSet creates more EphemeralRunners than `spec.replicas` on scale up (stale cache, no create expectations)対応中かも @yaronr が 29 日前に担当しました。 オープン
難易度 4/5 3〜5日 初心者へのやさしさ 55/100
actions/actions-runner-controller#4722 · コメント 1 件 ·
メンテナーはふだん 1 日以内に返信
actions/actions-runner-controller の issue をすべて見る
似ている issue
-
難易度 2/5 1〜3時間 初心者へのやさしさ 82/100
-
needs discussion
難易度 2/5 1〜3時間 初心者へのやさしさ 78/100
openfga/openfga.dev#1436 · コメント 1 件 ·
メンテナーはふだん 2 日以内に返信
-
難易度 2/5 1〜3時間 初心者へのやさしさ 75/100
digital-go-jp/abr-geocoder#457 ·
メンテナーはふだん 3 日以内に返信
-
benchmark
難易度 1/5 1〜3時間 初心者へのやさしさ 72/100
メンテナーはふだん 1 日以内に返信
-
難易度 2/5 1〜3時間 初心者へのやさしさ 77/100
mozilla-ai/cq#601 ·
メンテナーはふだん 1 日以内に返信