Controller upgrade cancels jobs that started shortly before their runner set is deleted

オープン
#4,736 コメント 2 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

@ankit373 がすでに取り組んでいます。

2026年11月10日 から。

評価

難易度
4/5
見積もり時間
3〜5日
初心者へのやさしさ
45/100
issue の種類
バグ
明瞭さ
明確に書かれている
活発さ
活発
技術スタック
github-actions, go, kubernetes
領域
backend, devops

調査の方向性

Start by tracing the scale set cleanup path that calls RemoveRunner and checks HasJob(), then review the ephemeral runner status update path that records status.jobId. Reproduce the timing window described during a controller upgrade and inspect the cleanup tests for busy runners. Done means a runner executing a job is preserved during scale set deletion, with a regression test covering a registered runner whose job ID has not yet been recorded.

索引モデルが issue の本文から書いたものです。

説明

Checks
  • I've already read the troubleshooting guide and I'm sure my issue is not covered in it.
  • I am using charts that are officially provided
Controller Version

0.15.1 (upgraded from 0.14.2)

Deployment Method

Kustomize (manifests rendered from the official charts), applied with kubectl apply --server-side; ArgoCD with selfHeal on stage and prod.

Checks
  • This isn't a question or user support case.
  • I've read the Changelog before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes
To Reproduce
  1. Run 0.14.2 with a scale set (minRunners: 0, maxRunners: 20).
  2. Dispatch jobs onto the scale set every 10 seconds.
  3. While jobs keep starting, upgrade the controller (CRDs and controller Deployment) to 0.15.1. The new controller deletes the 0.14.2 scale sets, since their major.minor differs.
  4. Check the conclusion of the jobs that started in the seconds before their runner was deleted.
Describe the bug

A job that starts on a runner whose EphemeralRunner has no status.jobId yet is cancelled when the scale set is deleted. The controller does not see the job (HasJob() is false), calls RemoveRunner, the service accepts the removal although the runner is executing the job, and the pod is deleted. The job fails with Process completed with exit code 130. and The runner has received a shutdown signal.

Runners whose status.jobId was already recorded are skipped (Skipping ephemeral runner since it is running a job) and their jobs finish.

Describe the expected behavior

A runner executing a job is not removed while its scale set is deleted. Either RemoveRunner refuses a busy runner (JobStillRunningError), as the controller expects, or the controller treats a registered runner without a recorded job as possibly busy until the service confirms otherwise.

Additional Context

Related: #4668 states that a late or lost job patch only costs a wasted API call, since RemoveRunner is rejected with JobStillRunningError while the job runs; here the service accepted it. #4683 protects runners that have no runner ID yet; the cancelled runners had one. #3298 reported the same symptom on 0.8.2 without an upgrade.

Measured on a dev cluster (EKS) with a 0.14.2 to 0.15.1 upgrade, 300 and 900 second jobs on one scale set, three runs:

Run Jobs running when their scale set was deleted With status.jobId at that point Cancelled Job age when its runner was removed
1 4 4 0 n/a
2 1 0 1 54s
3 13 12 1 9s

In run 2 the controller was restarted 21 seconds after the job started; in run 3 the job started 23 seconds after the new controller was applied, while the old listeners were being replaced. In both cases the runner had a runnerId but no jobId when the controller cleaned up the scale set.

Run 3, controller log for the cancelled runner (names shortened):

14:43:46Z INFO EphemeralRunner     Updated ephemeral runner status
14:43:56Z INFO EphemeralRunnerSet  Cleaning up the ephemeral runner
14:43:57Z INFO EphemeralRunnerSet  Deleting ephemeral runner after removing from the service  {"runnerId": ...}
14:43:57Z INFO EphemeralRunnerSet  Deleted ephemeral runner

The job (started_at 14:43:48Z) failed at 14:43:58Z with exit code 130. In the same reconcile, the 12 runners with a recorded jobId were skipped and their jobs succeeded, 8 of them started during the swap as well.

Controller Logs
Run 3: controller log for the cancelled runner (8pjml), a runner that was skipped (77qj7) and their EphemeralRunnerSet
2026-11-05T14:43:46Z	INFO	EphemeralRunner	Runner container is still running; updating ephemeral runner status	{"version": "0.15.1", "ephemeralrunner": {"name":"infra-aws-core-2-default-rts7j-runner-77qj7","namespace":"infra"}}
2026-11-05T14:43:46Z	INFO	EphemeralRunner	Runner container is still running; updating ephemeral runner status	{"version": "0.15.1", "ephemeralrunner": {"name":"infra-aws-core-2-default-rts7j-runner-8pjml","namespace":"infra"}}
2026-11-05T14:43:46Z	INFO	EphemeralRunner	Updated ephemeral runner status	{"version": "0.15.1", "ephemeralrunner": {"name":"infra-aws-core-2-default-rts7j-runner-8pjml","namespace":"infra"}}
2026-11-05T14:43:46Z	INFO	EphemeralRunner	Updating ephemeral runner status	{"version": "0.15.1", "ephemeralrunner": {"name":"infra-aws-core-2-default-rts7j-runner-8pjml","namespace":"infra"}, "statusPhase": "Running", "statusReason": "", "statusMessage": "", "ready": true}
2026-11-05T14:43:56Z	INFO	EphemeralRunnerSet	Clean up runner counts	{"version": "0.15.1", "ephemeralrunnerset": {"name":"infra-aws-core-2-default-rts7j","namespace":"infra"}, "pending": 1, "running": 12, "finished": 0, "failed": 0, "deleting": 0, "outdated": 0}
2026-11-05T14:43:56Z	INFO	EphemeralRunnerSet	Cleaning up the ephemeral runner	{"version": "0.15.1", "ephemeralrunnerset": {"name":"infra-aws-core-2-default-rts7j","namespace":"infra"}, "name": "infra-aws-core-2-default-rts7j-runner-8pjml"}
2026-11-05T14:43:56Z	INFO	EphemeralRunnerSet	Cleanup pending or running ephemeral runners	{"version": "0.15.1", "ephemeralrunnerset": {"name":"infra-aws-core-2-default-rts7j","namespace":"infra"}}
2026-11-05T14:43:56Z	INFO	EphemeralRunnerSet	Deleting resources	{"version": "0.15.1", "ephemeralrunnerset": {"name":"infra-aws-core-2-default-rts7j","namespace":"infra"}}
2026-11-05T14:43:57Z	ERROR	EphemeralRunner	Failed to update with finalizer set	{"version": "0.15.1", "ephemeralrunner": {"name":"infra-aws-core-2-default-rts7j-runner-8pjml","namespace":"infra"}, "error": "EphemeralRunner.actions.github.com \"infra-aws-core-2-default-rts7j-runner-8pjml\" is invalid: metadata.finalizers: Forbidden: no new finalizers can be added if the object is being deleted, found new finalizers []string{\"ephemeralrunner.actions.github.com/runner-registration-finalizer\"}"}
2026-11-05T14:43:57Z	ERROR	Reconciler error	{"controller": "ephemeralrunner", "controllerGroup": "actions.github.com", "controllerKind": "EphemeralRunner", "EphemeralRunner": {"name":"infra-aws-core-2-default-rts7j-runner-8pjml","namespace":"infra"}, "namespace": "infra", "name": "infra-aws-core-2-default-rts7j-runner-8pjml", "reconcileID": "fc326ee9-1604-4f13-bdd9-e02d853ac08b", "error": "EphemeralRunner.actions.github.com \"infra-aws-core-2-default-rts7j-runner-8pjml\" is invalid: metadata.finalizers: Forbidden: no new finalizers can be added if the object is being deleted, found new finalizers []string{\"ephemeralrunner.actions.github.com/runner-registration-finalizer\"}"}
2026-11-05T14:43:57Z	INFO	EphemeralRunner	Adding finalizers	{"version": "0.15.1", "ephemeralrunner": {"name":"infra-aws-core-2-default-rts7j-runner-8pjml","namespace":"infra"}}
2026-11-05T14:43:57Z	INFO	EphemeralRunner	Cleaning up the runner jitconfig secret	{"version": "0.15.1", "ephemeralrunner": {"name":"infra-aws-core-2-default-rts7j-runner-8pjml","namespace":"infra"}}
2026-11-05T14:43:57Z	INFO	EphemeralRunner	Cleaning up the runner pod	{"version": "0.15.1", "ephemeralrunner": {"name":"infra-aws-core-2-default-rts7j-runner-8pjml","namespace":"infra"}}
2026-11-05T14:43:57Z	INFO	EphemeralRunner	Deleted jitconfig secret	{"version": "0.15.1", "ephemeralrunner": {"name":"infra-aws-core-2-default-rts7j-runner-8pjml","namespace":"infra"}}
2026-11-05T14:43:57Z	INFO	EphemeralRunner	Deleted the runner pod	{"version": "0.15.1", "ephemeralrunner": {"name":"infra-aws-core-2-default-rts7j-runner-8pjml","namespace":"infra"}}
2026-11-05T14:43:57Z	INFO	EphemeralRunner	Deleting the jitconfig secret	{"version": "0.15.1", "ephemeralrunner": {"name":"infra-aws-core-2-default-rts7j-runner-8pjml","namespace":"infra"}}
2026-11-05T14:43:57Z	INFO	EphemeralRunner	Deleting the runner pod	{"version": "0.15.1", "ephemeralrunner": {"name":"infra-aws-core-2-default-rts7j-runner-8pjml","namespace":"infra"}}
2026-11-05T14:43:57Z	INFO	EphemeralRunner	Finalizing ephemeral runner	{"version": "0.15.1", "ephemeralrunner": {"name":"infra-aws-core-2-default-rts7j-runner-8pjml","namespace":"infra"}}
2026-11-05T14:43:57Z	INFO	EphemeralRunner	Removed finalizer from ephemeral runner	{"version": "0.15.1", "ephemeralrunner": {"name":"infra-aws-core-2-default-rts7j-runner-8pjml","namespace":"infra"}}
2026-11-05T14:43:57Z	INFO	EphemeralRunner	Removing finalizer	{"version": "0.15.1", "ephemeralrunner": {"name":"infra-aws-core-2-default-rts7j-runner-8pjml","namespace":"infra"}}
2026-11-05T14:43:57Z	INFO	EphemeralRunnerSet	Deleted ephemeral runner	{"version": "0.15.1", "ephemeralrunnerset": {"name":"infra-aws-core-2-default-rts7j","namespace":"infra"}, "name": "infra-aws-core-2-default-rts7j-runner-8pjml", "runnerId": 10721544}
2026-11-05T14:43:57Z	INFO	EphemeralRunnerSet	Deleting ephemeral runner after removing from the service	{"version": "0.15.1", "ephemeralrunnerset": {"name":"infra-aws-core-2-default-rts7j","namespace":"infra"}, "name": "infra-aws-core-2-default-rts7j-runner-8pjml", "runnerId": 10721544}
2026-11-05T14:43:57Z	INFO	EphemeralRunnerSet	Deleting resources	{"version": "0.15.1", "ephemeralrunnerset": {"name":"infra-aws-core-2-default-rts7j","namespace":"infra"}}
2026-11-05T14:43:57Z	INFO	EphemeralRunnerSet	Skipping ephemeral runner since it is running a job	{"version": "0.15.1", "ephemeralrunnerset": {"name":"infra-aws-core-2-default-rts7j","namespace":"infra"}, "name": "infra-aws-core-2-default-rts7j-runner-77qj7", "workflowRunId": 42047428183, "jobId": "3fa9d8e3-1d97-5989-9d6e-e4038410b5ba"}
2026-11-05T14:43:57Z	INFO	EphemeralRunnerSet	Skipping ephemeral runner since it is running a job	{"version": "0.15.1", "ephemeralrunnerset": {"name":"infra-aws-core-2-default-rts7j","namespace":"infra"}, "name": "infra-aws-core-2-default-rts7j-runner-77qj7", "workflowRunId": 42047428183, "jobId": "3fa9d8e3-1d97-5989-9d6e-e4038410b5ba"}
2026-11-05T14:43:57Z	INFO	EphemeralRunnerSet	Skipping ephemeral runner since it is running a job	{"version": "0.15.1", "ephemeralrunnerset": {"name":"infra-aws-core-2-default-rts7j","namespace":"infra"}, "name": "infra-aws-core-2-default-rts7j-runner-77qj7", "workflowRunId": 42047428183, "jobId": "3fa9d8e3-1d97-5989-9d6e-e4038410b5ba"}
2026-11-05T14:43:57Z	INFO	EphemeralRunnerSet	Skipping ephemeral runner since it is running a job	{"version": "0.15.1", "ephemeralrunnerset": {"name":"infra-aws-core-2-default-rts7j","namespace":"infra"}, "name": "infra-aws-core-2-default-rts7j-runner-77qj7", "workflowRunId": 42047428183, "jobId": "3fa9d8e3-1d97-5989-9d6e-e4038410b5ba"}
Runner Pod Logs
Run 3: job log of the cancelled job
2026-11-05T14:43:49.9103757Z Current runner version: '2.338.1'
2026-11-05T14:43:49.9114558Z Runner name: 'infra-aws-core-2-default-rts7j-runner-8pjml'
2026-11-05T14:43:49.9116421Z Runner group name: 'infra-dev-self-hosted-runners'
2026-11-05T14:43:49.9118353Z Machine name: 'infra-aws-core-2-default-rts7j-runner-8pjml'
2026-11-05T14:43:49.9124877Z ##[group]GITHUB_TOKEN Permissions
2026-11-05T14:43:49.9128855Z Metadata: read
2026-11-05T14:43:49.9129771Z ##[endgroup]
2026-11-05T14:43:49.9133335Z Secret source: Actions
2026-11-05T14:43:49.9134775Z Cache mode: write
2026-11-05T14:43:49.9136019Z Prepare workflow directory
2026-11-05T14:43:49.9691728Z Prepare all required actions
2026-11-05T14:43:49.9867996Z Complete job name: test (dev-aws-core-2-default)
2026-11-05T14:43:50.1409078Z A job started hook has been configured by the self-hosted runner administrator
2026-11-05T14:43:50.1669201Z ##[group]Run '/home/runner/hooks/label-runner-pod.sh'
2026-11-05T14:43:50.1689061Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2026-11-05T14:43:50.1691119Z ##[endgroup]
2026-11-05T14:43:50.6453779Z Client Version: v1.35.8
2026-11-05T14:43:50.6455320Z Kustomize Version: v5.7.1
2026-11-05T14:43:50.6456952Z Server Version: v1.36.4-eks-cfb47f5
2026-11-05T14:43:50.6459249Z Adding label 'camunda.cloud/workflow=Test-dev-self-hosted-runners' to the runner
2026-11-05T14:43:50.6647463Z Adding label 'camunda.cloud/workflow-repository=camunda-infra-core' to the runner
2026-11-05T14:43:50.8239381Z Adding label 'camunda.cloud/job-id=test' to the runner
2026-11-05T14:43:50.9302870Z Adding label 'camunda.cloud/run-id=42047639016' to the runner
2026-11-05T14:43:51.0760490Z Adding label 'camunda.cloud/run-attempt=1' to the runner
2026-11-05T14:43:51.1591572Z Adding label 'camunda.cloud/github-ref=heads-stage' to the runner
2026-11-05T14:43:51.3887508Z Adding label 'camunda.cloud/base-ref=not-applicable' to the runner
2026-11-05T14:43:51.5314290Z ##[group]Run # The runner name tells which fleet picked the job up (maint-* for the maintenance sets).
2026-11-05T14:43:51.5317635Z # The runner name tells which fleet picked the job up (maint-* for the maintenance sets).
2026-11-05T14:43:51.5320322Z echo "Running on ${RUNNER_NAME} (label ${LABEL}) at $(date -u +%FT%TZ)"
2026-11-05T14:43:51.5323417Z sleep "$SLEEP_SECONDS"
2026-11-05T14:43:51.5324700Z echo "Finished at $(date -u +%FT%TZ)"
2026-11-05T14:43:51.5339535Z shell: /usr/bin/bash -e {0}
2026-11-05T14:43:51.5340790Z env:
2026-11-05T14:43:51.5341730Z   SLEEP_SECONDS: 900
2026-11-05T14:43:51.5342831Z   LABEL: dev-aws-core-2-default
2026-11-05T14:43:51.5343908Z ##[endgroup]
2026-11-05T14:43:51.5419819Z Running on infra-aws-core-2-default-rts7j-runner-8pjml (label dev-aws-core-2-default) at 2026-11-05T14:43:51Z
2026-11-05T14:43:57.1238858Z ##[error]Process completed with exit code 130.
2026-11-05T14:43:57.1424296Z Cleaning up orphan processes
2026-11-05T14:43:57.1555198Z ##[error]The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.
主要言語
Go
スター
6.7k
フォーク
1.5k
平均マージ
1時間 34分
マージ済み PR(30日)
5

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

actions/actions-runner-controller のほかの issue

actions/actions-runner-controller の issue をすべて見る

似ている issue

Go の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。