Controller upgrade cancels jobs that started shortly before their runner set is deleted
Maintainers usually reply within 1 day
@ankit373 is already working on this.
Since Nov 10, 2026.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 45/100
Research direction
Start by tracing the scale set cleanup path that calls RemoveRunner and checks HasJob(), then review the ephemeral runner status update path that records status.jobId. Reproduce the timing window described during a controller upgrade and inspect the cleanup tests for busy runners. Done means a runner executing a job is preserved during scale set deletion, with a regression test covering a registered runner whose job ID has not yet been recorded.
Written by the indexing model from the issue text.
Description
Checks
- I've already read the troubleshooting guide and I'm sure my issue is not covered in it.
- I am using charts that are officially provided
Controller Version
0.15.1 (upgraded from 0.14.2)
Deployment Method
Kustomize (manifests rendered from the official charts), applied with kubectl apply --server-side; ArgoCD with selfHeal on stage and prod.
Checks
- This isn't a question or user support case.
- I've read the Changelog before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes
To Reproduce
- Run 0.14.2 with a scale set (
minRunners: 0,maxRunners: 20). - Dispatch jobs onto the scale set every 10 seconds.
- While jobs keep starting, upgrade the controller (CRDs and controller Deployment) to 0.15.1. The new controller deletes the 0.14.2 scale sets, since their
major.minordiffers. - Check the conclusion of the jobs that started in the seconds before their runner was deleted.
Describe the bug
A job that starts on a runner whose EphemeralRunner has no status.jobId yet is cancelled when the scale set is deleted. The controller does not see the job (HasJob() is false), calls RemoveRunner, the service accepts the removal although the runner is executing the job, and the pod is deleted. The job fails with Process completed with exit code 130. and The runner has received a shutdown signal.
Runners whose status.jobId was already recorded are skipped (Skipping ephemeral runner since it is running a job) and their jobs finish.
Describe the expected behavior
A runner executing a job is not removed while its scale set is deleted. Either RemoveRunner refuses a busy runner (JobStillRunningError), as the controller expects, or the controller treats a registered runner without a recorded job as possibly busy until the service confirms otherwise.
Additional Context
Related: #4668 states that a late or lost job patch only costs a wasted API call, since RemoveRunner is rejected with JobStillRunningError while the job runs; here the service accepted it. #4683 protects runners that have no runner ID yet; the cancelled runners had one. #3298 reported the same symptom on 0.8.2 without an upgrade.
Measured on a dev cluster (EKS) with a 0.14.2 to 0.15.1 upgrade, 300 and 900 second jobs on one scale set, three runs:
| Run | Jobs running when their scale set was deleted | With status.jobId at that point |
Cancelled | Job age when its runner was removed |
|---|---|---|---|---|
| 1 | 4 | 4 | 0 | n/a |
| 2 | 1 | 0 | 1 | 54s |
| 3 | 13 | 12 | 1 | 9s |
In run 2 the controller was restarted 21 seconds after the job started; in run 3 the job started 23 seconds after the new controller was applied, while the old listeners were being replaced. In both cases the runner had a runnerId but no jobId when the controller cleaned up the scale set.
Run 3, controller log for the cancelled runner (names shortened):
14:43:46Z INFO EphemeralRunner Updated ephemeral runner status
14:43:56Z INFO EphemeralRunnerSet Cleaning up the ephemeral runner
14:43:57Z INFO EphemeralRunnerSet Deleting ephemeral runner after removing from the service {"runnerId": ...}
14:43:57Z INFO EphemeralRunnerSet Deleted ephemeral runner
The job (started_at 14:43:48Z) failed at 14:43:58Z with exit code 130. In the same reconcile, the 12 runners with a recorded jobId were skipped and their jobs succeeded, 8 of them started during the swap as well.
Controller Logs
Run 3: controller log for the cancelled runner (8pjml), a runner that was skipped (77qj7) and their EphemeralRunnerSet
2026-11-05T14:43:46Z INFO EphemeralRunner Runner container is still running; updating ephemeral runner status {"version": "0.15.1", "ephemeralrunner": {"name":"infra-aws-core-2-default-rts7j-runner-77qj7","namespace":"infra"}}
2026-11-05T14:43:46Z INFO EphemeralRunner Runner container is still running; updating ephemeral runner status {"version": "0.15.1", "ephemeralrunner": {"name":"infra-aws-core-2-default-rts7j-runner-8pjml","namespace":"infra"}}
2026-11-05T14:43:46Z INFO EphemeralRunner Updated ephemeral runner status {"version": "0.15.1", "ephemeralrunner": {"name":"infra-aws-core-2-default-rts7j-runner-8pjml","namespace":"infra"}}
2026-11-05T14:43:46Z INFO EphemeralRunner Updating ephemeral runner status {"version": "0.15.1", "ephemeralrunner": {"name":"infra-aws-core-2-default-rts7j-runner-8pjml","namespace":"infra"}, "statusPhase": "Running", "statusReason": "", "statusMessage": "", "ready": true}
2026-11-05T14:43:56Z INFO EphemeralRunnerSet Clean up runner counts {"version": "0.15.1", "ephemeralrunnerset": {"name":"infra-aws-core-2-default-rts7j","namespace":"infra"}, "pending": 1, "running": 12, "finished": 0, "failed": 0, "deleting": 0, "outdated": 0}
2026-11-05T14:43:56Z INFO EphemeralRunnerSet Cleaning up the ephemeral runner {"version": "0.15.1", "ephemeralrunnerset": {"name":"infra-aws-core-2-default-rts7j","namespace":"infra"}, "name": "infra-aws-core-2-default-rts7j-runner-8pjml"}
2026-11-05T14:43:56Z INFO EphemeralRunnerSet Cleanup pending or running ephemeral runners {"version": "0.15.1", "ephemeralrunnerset": {"name":"infra-aws-core-2-default-rts7j","namespace":"infra"}}
2026-11-05T14:43:56Z INFO EphemeralRunnerSet Deleting resources {"version": "0.15.1", "ephemeralrunnerset": {"name":"infra-aws-core-2-default-rts7j","namespace":"infra"}}
2026-11-05T14:43:57Z ERROR EphemeralRunner Failed to update with finalizer set {"version": "0.15.1", "ephemeralrunner": {"name":"infra-aws-core-2-default-rts7j-runner-8pjml","namespace":"infra"}, "error": "EphemeralRunner.actions.github.com \"infra-aws-core-2-default-rts7j-runner-8pjml\" is invalid: metadata.finalizers: Forbidden: no new finalizers can be added if the object is being deleted, found new finalizers []string{\"ephemeralrunner.actions.github.com/runner-registration-finalizer\"}"}
2026-11-05T14:43:57Z ERROR Reconciler error {"controller": "ephemeralrunner", "controllerGroup": "actions.github.com", "controllerKind": "EphemeralRunner", "EphemeralRunner": {"name":"infra-aws-core-2-default-rts7j-runner-8pjml","namespace":"infra"}, "namespace": "infra", "name": "infra-aws-core-2-default-rts7j-runner-8pjml", "reconcileID": "fc326ee9-1604-4f13-bdd9-e02d853ac08b", "error": "EphemeralRunner.actions.github.com \"infra-aws-core-2-default-rts7j-runner-8pjml\" is invalid: metadata.finalizers: Forbidden: no new finalizers can be added if the object is being deleted, found new finalizers []string{\"ephemeralrunner.actions.github.com/runner-registration-finalizer\"}"}
2026-11-05T14:43:57Z INFO EphemeralRunner Adding finalizers {"version": "0.15.1", "ephemeralrunner": {"name":"infra-aws-core-2-default-rts7j-runner-8pjml","namespace":"infra"}}
2026-11-05T14:43:57Z INFO EphemeralRunner Cleaning up the runner jitconfig secret {"version": "0.15.1", "ephemeralrunner": {"name":"infra-aws-core-2-default-rts7j-runner-8pjml","namespace":"infra"}}
2026-11-05T14:43:57Z INFO EphemeralRunner Cleaning up the runner pod {"version": "0.15.1", "ephemeralrunner": {"name":"infra-aws-core-2-default-rts7j-runner-8pjml","namespace":"infra"}}
2026-11-05T14:43:57Z INFO EphemeralRunner Deleted jitconfig secret {"version": "0.15.1", "ephemeralrunner": {"name":"infra-aws-core-2-default-rts7j-runner-8pjml","namespace":"infra"}}
2026-11-05T14:43:57Z INFO EphemeralRunner Deleted the runner pod {"version": "0.15.1", "ephemeralrunner": {"name":"infra-aws-core-2-default-rts7j-runner-8pjml","namespace":"infra"}}
2026-11-05T14:43:57Z INFO EphemeralRunner Deleting the jitconfig secret {"version": "0.15.1", "ephemeralrunner": {"name":"infra-aws-core-2-default-rts7j-runner-8pjml","namespace":"infra"}}
2026-11-05T14:43:57Z INFO EphemeralRunner Deleting the runner pod {"version": "0.15.1", "ephemeralrunner": {"name":"infra-aws-core-2-default-rts7j-runner-8pjml","namespace":"infra"}}
2026-11-05T14:43:57Z INFO EphemeralRunner Finalizing ephemeral runner {"version": "0.15.1", "ephemeralrunner": {"name":"infra-aws-core-2-default-rts7j-runner-8pjml","namespace":"infra"}}
2026-11-05T14:43:57Z INFO EphemeralRunner Removed finalizer from ephemeral runner {"version": "0.15.1", "ephemeralrunner": {"name":"infra-aws-core-2-default-rts7j-runner-8pjml","namespace":"infra"}}
2026-11-05T14:43:57Z INFO EphemeralRunner Removing finalizer {"version": "0.15.1", "ephemeralrunner": {"name":"infra-aws-core-2-default-rts7j-runner-8pjml","namespace":"infra"}}
2026-11-05T14:43:57Z INFO EphemeralRunnerSet Deleted ephemeral runner {"version": "0.15.1", "ephemeralrunnerset": {"name":"infra-aws-core-2-default-rts7j","namespace":"infra"}, "name": "infra-aws-core-2-default-rts7j-runner-8pjml", "runnerId": 10721544}
2026-11-05T14:43:57Z INFO EphemeralRunnerSet Deleting ephemeral runner after removing from the service {"version": "0.15.1", "ephemeralrunnerset": {"name":"infra-aws-core-2-default-rts7j","namespace":"infra"}, "name": "infra-aws-core-2-default-rts7j-runner-8pjml", "runnerId": 10721544}
2026-11-05T14:43:57Z INFO EphemeralRunnerSet Deleting resources {"version": "0.15.1", "ephemeralrunnerset": {"name":"infra-aws-core-2-default-rts7j","namespace":"infra"}}
2026-11-05T14:43:57Z INFO EphemeralRunnerSet Skipping ephemeral runner since it is running a job {"version": "0.15.1", "ephemeralrunnerset": {"name":"infra-aws-core-2-default-rts7j","namespace":"infra"}, "name": "infra-aws-core-2-default-rts7j-runner-77qj7", "workflowRunId": 42047428183, "jobId": "3fa9d8e3-1d97-5989-9d6e-e4038410b5ba"}
2026-11-05T14:43:57Z INFO EphemeralRunnerSet Skipping ephemeral runner since it is running a job {"version": "0.15.1", "ephemeralrunnerset": {"name":"infra-aws-core-2-default-rts7j","namespace":"infra"}, "name": "infra-aws-core-2-default-rts7j-runner-77qj7", "workflowRunId": 42047428183, "jobId": "3fa9d8e3-1d97-5989-9d6e-e4038410b5ba"}
2026-11-05T14:43:57Z INFO EphemeralRunnerSet Skipping ephemeral runner since it is running a job {"version": "0.15.1", "ephemeralrunnerset": {"name":"infra-aws-core-2-default-rts7j","namespace":"infra"}, "name": "infra-aws-core-2-default-rts7j-runner-77qj7", "workflowRunId": 42047428183, "jobId": "3fa9d8e3-1d97-5989-9d6e-e4038410b5ba"}
2026-11-05T14:43:57Z INFO EphemeralRunnerSet Skipping ephemeral runner since it is running a job {"version": "0.15.1", "ephemeralrunnerset": {"name":"infra-aws-core-2-default-rts7j","namespace":"infra"}, "name": "infra-aws-core-2-default-rts7j-runner-77qj7", "workflowRunId": 42047428183, "jobId": "3fa9d8e3-1d97-5989-9d6e-e4038410b5ba"}
Runner Pod Logs
Run 3: job log of the cancelled job
2026-11-05T14:43:49.9103757Z Current runner version: '2.338.1'
2026-11-05T14:43:49.9114558Z Runner name: 'infra-aws-core-2-default-rts7j-runner-8pjml'
2026-11-05T14:43:49.9116421Z Runner group name: 'infra-dev-self-hosted-runners'
2026-11-05T14:43:49.9118353Z Machine name: 'infra-aws-core-2-default-rts7j-runner-8pjml'
2026-11-05T14:43:49.9124877Z ##[group]GITHUB_TOKEN Permissions
2026-11-05T14:43:49.9128855Z Metadata: read
2026-11-05T14:43:49.9129771Z ##[endgroup]
2026-11-05T14:43:49.9133335Z Secret source: Actions
2026-11-05T14:43:49.9134775Z Cache mode: write
2026-11-05T14:43:49.9136019Z Prepare workflow directory
2026-11-05T14:43:49.9691728Z Prepare all required actions
2026-11-05T14:43:49.9867996Z Complete job name: test (dev-aws-core-2-default)
2026-11-05T14:43:50.1409078Z A job started hook has been configured by the self-hosted runner administrator
2026-11-05T14:43:50.1669201Z ##[group]Run '/home/runner/hooks/label-runner-pod.sh'
2026-11-05T14:43:50.1689061Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2026-11-05T14:43:50.1691119Z ##[endgroup]
2026-11-05T14:43:50.6453779Z Client Version: v1.35.8
2026-11-05T14:43:50.6455320Z Kustomize Version: v5.7.1
2026-11-05T14:43:50.6456952Z Server Version: v1.36.4-eks-cfb47f5
2026-11-05T14:43:50.6459249Z Adding label 'camunda.cloud/workflow=Test-dev-self-hosted-runners' to the runner
2026-11-05T14:43:50.6647463Z Adding label 'camunda.cloud/workflow-repository=camunda-infra-core' to the runner
2026-11-05T14:43:50.8239381Z Adding label 'camunda.cloud/job-id=test' to the runner
2026-11-05T14:43:50.9302870Z Adding label 'camunda.cloud/run-id=42047639016' to the runner
2026-11-05T14:43:51.0760490Z Adding label 'camunda.cloud/run-attempt=1' to the runner
2026-11-05T14:43:51.1591572Z Adding label 'camunda.cloud/github-ref=heads-stage' to the runner
2026-11-05T14:43:51.3887508Z Adding label 'camunda.cloud/base-ref=not-applicable' to the runner
2026-11-05T14:43:51.5314290Z ##[group]Run # The runner name tells which fleet picked the job up (maint-* for the maintenance sets).
2026-11-05T14:43:51.5317635Z # The runner name tells which fleet picked the job up (maint-* for the maintenance sets).
2026-11-05T14:43:51.5320322Z echo "Running on ${RUNNER_NAME} (label ${LABEL}) at $(date -u +%FT%TZ)"
2026-11-05T14:43:51.5323417Z sleep "$SLEEP_SECONDS"
2026-11-05T14:43:51.5324700Z echo "Finished at $(date -u +%FT%TZ)"
2026-11-05T14:43:51.5339535Z shell: /usr/bin/bash -e {0}
2026-11-05T14:43:51.5340790Z env:
2026-11-05T14:43:51.5341730Z SLEEP_SECONDS: 900
2026-11-05T14:43:51.5342831Z LABEL: dev-aws-core-2-default
2026-11-05T14:43:51.5343908Z ##[endgroup]
2026-11-05T14:43:51.5419819Z Running on infra-aws-core-2-default-rts7j-runner-8pjml (label dev-aws-core-2-default) at 2026-11-05T14:43:51Z
2026-11-05T14:43:57.1238858Z ##[error]Process completed with exit code 130.
2026-11-05T14:43:57.1424296Z Cleaning up orphan processes
2026-11-05T14:43:57.1555198Z ##[error]The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.
- Dominant language
- Go
- Stars
- 6.7k
- Forks
- 1.5k
- Avg merge
- 1h 34m
- Merged PRs (30d)
- 5
Getting set up
- Ships a Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from actions/actions-runner-controller
-
community enhancement needs triage
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
actions/actions-runner-controller#3997 · 9 comments · 27 reactions ·
Maintainers usually reply within 1 day
-
Difficulty 4/5 3-5 days Newbie friendliness 48/100
actions/actions-runner-controller#4739 · 1 comment · 4 reactions ·
Maintainers usually reply within 1 day
-
gha-runner-scale-set: support Google Cloud Secret Manager as a vault for GitHub credentialsPossibly taken A pull request linked to this issue is open or already merged. Opencommunity enhancement needs triage
Difficulty 4/5 3-5 days Newbie friendliness 52/100
actions/actions-runner-controller#4737 · 1 comment ·
Maintainers usually reply within 1 day
-
EphemeralRunnerSet controller creates more runners than `spec.replicas` when its cache lags its own createsPossibly taken @jia-pylon claimed this 14 days ago. Open
Difficulty 4/5 3-5 days Newbie friendliness 30/100
actions/actions-runner-controller#4734 · 1 comment ·
Maintainers usually reply within 1 day
-
EphemeralRunnerSet creates more EphemeralRunners than `spec.replicas` on scale up (stale cache, no create expectations)Possibly taken @yaronr claimed this 29 days ago. Open
Difficulty 4/5 3-5 days Newbie friendliness 55/100
actions/actions-runner-controller#4722 · 1 comment ·
Maintainers usually reply within 1 day
All issues in actions/actions-runner-controller
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day
-
Inconsistency between golang and cue in specPossibly taken A pull request linked to this issue is open or already merged. Open
Difficulty 1/5 Under an hour Newbie friendliness 88/100
Maintainers usually reply within 1 day
-
security
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 73/100
Tencent/LoopForge#1087 · 1 comment ·
Maintainers usually reply within 1 day