Controller upgrade cancels jobs that started shortly before their runner set is deleted

Open
#4,736 2 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

@ankit373 is already working on this.

Since Nov 10, 2026.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
45/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
github-actions, go, kubernetes
Domain
backend, devops

Research direction

Start by tracing the scale set cleanup path that calls RemoveRunner and checks HasJob(), then review the ephemeral runner status update path that records status.jobId. Reproduce the timing window described during a controller upgrade and inspect the cleanup tests for busy runners. Done means a runner executing a job is preserved during scale set deletion, with a regression test covering a registered runner whose job ID has not yet been recorded.

Written by the indexing model from the issue text.

Description

Checks
  • I've already read the troubleshooting guide and I'm sure my issue is not covered in it.
  • I am using charts that are officially provided
Controller Version

0.15.1 (upgraded from 0.14.2)

Deployment Method

Kustomize (manifests rendered from the official charts), applied with kubectl apply --server-side; ArgoCD with selfHeal on stage and prod.

Checks
  • This isn't a question or user support case.
  • I've read the Changelog before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes
To Reproduce
  1. Run 0.14.2 with a scale set (minRunners: 0, maxRunners: 20).
  2. Dispatch jobs onto the scale set every 10 seconds.
  3. While jobs keep starting, upgrade the controller (CRDs and controller Deployment) to 0.15.1. The new controller deletes the 0.14.2 scale sets, since their major.minor differs.
  4. Check the conclusion of the jobs that started in the seconds before their runner was deleted.
Describe the bug

A job that starts on a runner whose EphemeralRunner has no status.jobId yet is cancelled when the scale set is deleted. The controller does not see the job (HasJob() is false), calls RemoveRunner, the service accepts the removal although the runner is executing the job, and the pod is deleted. The job fails with Process completed with exit code 130. and The runner has received a shutdown signal.

Runners whose status.jobId was already recorded are skipped (Skipping ephemeral runner since it is running a job) and their jobs finish.

Describe the expected behavior

A runner executing a job is not removed while its scale set is deleted. Either RemoveRunner refuses a busy runner (JobStillRunningError), as the controller expects, or the controller treats a registered runner without a recorded job as possibly busy until the service confirms otherwise.

Additional Context

Related: #4668 states that a late or lost job patch only costs a wasted API call, since RemoveRunner is rejected with JobStillRunningError while the job runs; here the service accepted it. #4683 protects runners that have no runner ID yet; the cancelled runners had one. #3298 reported the same symptom on 0.8.2 without an upgrade.

Measured on a dev cluster (EKS) with a 0.14.2 to 0.15.1 upgrade, 300 and 900 second jobs on one scale set, three runs:

Run Jobs running when their scale set was deleted With status.jobId at that point Cancelled Job age when its runner was removed
1 4 4 0 n/a
2 1 0 1 54s
3 13 12 1 9s

In run 2 the controller was restarted 21 seconds after the job started; in run 3 the job started 23 seconds after the new controller was applied, while the old listeners were being replaced. In both cases the runner had a runnerId but no jobId when the controller cleaned up the scale set.

Run 3, controller log for the cancelled runner (names shortened):

14:43:46Z INFO EphemeralRunner     Updated ephemeral runner status
14:43:56Z INFO EphemeralRunnerSet  Cleaning up the ephemeral runner
14:43:57Z INFO EphemeralRunnerSet  Deleting ephemeral runner after removing from the service  {"runnerId": ...}
14:43:57Z INFO EphemeralRunnerSet  Deleted ephemeral runner

The job (started_at 14:43:48Z) failed at 14:43:58Z with exit code 130. In the same reconcile, the 12 runners with a recorded jobId were skipped and their jobs succeeded, 8 of them started during the swap as well.

Controller Logs
Run 3: controller log for the cancelled runner (8pjml), a runner that was skipped (77qj7) and their EphemeralRunnerSet
2026-11-05T14:43:46Z	INFO	EphemeralRunner	Runner container is still running; updating ephemeral runner status	{"version": "0.15.1", "ephemeralrunner": {"name":"infra-aws-core-2-default-rts7j-runner-77qj7","namespace":"infra"}}
2026-11-05T14:43:46Z	INFO	EphemeralRunner	Runner container is still running; updating ephemeral runner status	{"version": "0.15.1", "ephemeralrunner": {"name":"infra-aws-core-2-default-rts7j-runner-8pjml","namespace":"infra"}}
2026-11-05T14:43:46Z	INFO	EphemeralRunner	Updated ephemeral runner status	{"version": "0.15.1", "ephemeralrunner": {"name":"infra-aws-core-2-default-rts7j-runner-8pjml","namespace":"infra"}}
2026-11-05T14:43:46Z	INFO	EphemeralRunner	Updating ephemeral runner status	{"version": "0.15.1", "ephemeralrunner": {"name":"infra-aws-core-2-default-rts7j-runner-8pjml","namespace":"infra"}, "statusPhase": "Running", "statusReason": "", "statusMessage": "", "ready": true}
2026-11-05T14:43:56Z	INFO	EphemeralRunnerSet	Clean up runner counts	{"version": "0.15.1", "ephemeralrunnerset": {"name":"infra-aws-core-2-default-rts7j","namespace":"infra"}, "pending": 1, "running": 12, "finished": 0, "failed": 0, "deleting": 0, "outdated": 0}
2026-11-05T14:43:56Z	INFO	EphemeralRunnerSet	Cleaning up the ephemeral runner	{"version": "0.15.1", "ephemeralrunnerset": {"name":"infra-aws-core-2-default-rts7j","namespace":"infra"}, "name": "infra-aws-core-2-default-rts7j-runner-8pjml"}
2026-11-05T14:43:56Z	INFO	EphemeralRunnerSet	Cleanup pending or running ephemeral runners	{"version": "0.15.1", "ephemeralrunnerset": {"name":"infra-aws-core-2-default-rts7j","namespace":"infra"}}
2026-11-05T14:43:56Z	INFO	EphemeralRunnerSet	Deleting resources	{"version": "0.15.1", "ephemeralrunnerset": {"name":"infra-aws-core-2-default-rts7j","namespace":"infra"}}
2026-11-05T14:43:57Z	ERROR	EphemeralRunner	Failed to update with finalizer set	{"version": "0.15.1", "ephemeralrunner": {"name":"infra-aws-core-2-default-rts7j-runner-8pjml","namespace":"infra"}, "error": "EphemeralRunner.actions.github.com \"infra-aws-core-2-default-rts7j-runner-8pjml\" is invalid: metadata.finalizers: Forbidden: no new finalizers can be added if the object is being deleted, found new finalizers []string{\"ephemeralrunner.actions.github.com/runner-registration-finalizer\"}"}
2026-11-05T14:43:57Z	ERROR	Reconciler error	{"controller": "ephemeralrunner", "controllerGroup": "actions.github.com", "controllerKind": "EphemeralRunner", "EphemeralRunner": {"name":"infra-aws-core-2-default-rts7j-runner-8pjml","namespace":"infra"}, "namespace": "infra", "name": "infra-aws-core-2-default-rts7j-runner-8pjml", "reconcileID": "fc326ee9-1604-4f13-bdd9-e02d853ac08b", "error": "EphemeralRunner.actions.github.com \"infra-aws-core-2-default-rts7j-runner-8pjml\" is invalid: metadata.finalizers: Forbidden: no new finalizers can be added if the object is being deleted, found new finalizers []string{\"ephemeralrunner.actions.github.com/runner-registration-finalizer\"}"}
2026-11-05T14:43:57Z	INFO	EphemeralRunner	Adding finalizers	{"version": "0.15.1", "ephemeralrunner": {"name":"infra-aws-core-2-default-rts7j-runner-8pjml","namespace":"infra"}}
2026-11-05T14:43:57Z	INFO	EphemeralRunner	Cleaning up the runner jitconfig secret	{"version": "0.15.1", "ephemeralrunner": {"name":"infra-aws-core-2-default-rts7j-runner-8pjml","namespace":"infra"}}
2026-11-05T14:43:57Z	INFO	EphemeralRunner	Cleaning up the runner pod	{"version": "0.15.1", "ephemeralrunner": {"name":"infra-aws-core-2-default-rts7j-runner-8pjml","namespace":"infra"}}
2026-11-05T14:43:57Z	INFO	EphemeralRunner	Deleted jitconfig secret	{"version": "0.15.1", "ephemeralrunner": {"name":"infra-aws-core-2-default-rts7j-runner-8pjml","namespace":"infra"}}
2026-11-05T14:43:57Z	INFO	EphemeralRunner	Deleted the runner pod	{"version": "0.15.1", "ephemeralrunner": {"name":"infra-aws-core-2-default-rts7j-runner-8pjml","namespace":"infra"}}
2026-11-05T14:43:57Z	INFO	EphemeralRunner	Deleting the jitconfig secret	{"version": "0.15.1", "ephemeralrunner": {"name":"infra-aws-core-2-default-rts7j-runner-8pjml","namespace":"infra"}}
2026-11-05T14:43:57Z	INFO	EphemeralRunner	Deleting the runner pod	{"version": "0.15.1", "ephemeralrunner": {"name":"infra-aws-core-2-default-rts7j-runner-8pjml","namespace":"infra"}}
2026-11-05T14:43:57Z	INFO	EphemeralRunner	Finalizing ephemeral runner	{"version": "0.15.1", "ephemeralrunner": {"name":"infra-aws-core-2-default-rts7j-runner-8pjml","namespace":"infra"}}
2026-11-05T14:43:57Z	INFO	EphemeralRunner	Removed finalizer from ephemeral runner	{"version": "0.15.1", "ephemeralrunner": {"name":"infra-aws-core-2-default-rts7j-runner-8pjml","namespace":"infra"}}
2026-11-05T14:43:57Z	INFO	EphemeralRunner	Removing finalizer	{"version": "0.15.1", "ephemeralrunner": {"name":"infra-aws-core-2-default-rts7j-runner-8pjml","namespace":"infra"}}
2026-11-05T14:43:57Z	INFO	EphemeralRunnerSet	Deleted ephemeral runner	{"version": "0.15.1", "ephemeralrunnerset": {"name":"infra-aws-core-2-default-rts7j","namespace":"infra"}, "name": "infra-aws-core-2-default-rts7j-runner-8pjml", "runnerId": 10721544}
2026-11-05T14:43:57Z	INFO	EphemeralRunnerSet	Deleting ephemeral runner after removing from the service	{"version": "0.15.1", "ephemeralrunnerset": {"name":"infra-aws-core-2-default-rts7j","namespace":"infra"}, "name": "infra-aws-core-2-default-rts7j-runner-8pjml", "runnerId": 10721544}
2026-11-05T14:43:57Z	INFO	EphemeralRunnerSet	Deleting resources	{"version": "0.15.1", "ephemeralrunnerset": {"name":"infra-aws-core-2-default-rts7j","namespace":"infra"}}
2026-11-05T14:43:57Z	INFO	EphemeralRunnerSet	Skipping ephemeral runner since it is running a job	{"version": "0.15.1", "ephemeralrunnerset": {"name":"infra-aws-core-2-default-rts7j","namespace":"infra"}, "name": "infra-aws-core-2-default-rts7j-runner-77qj7", "workflowRunId": 42047428183, "jobId": "3fa9d8e3-1d97-5989-9d6e-e4038410b5ba"}
2026-11-05T14:43:57Z	INFO	EphemeralRunnerSet	Skipping ephemeral runner since it is running a job	{"version": "0.15.1", "ephemeralrunnerset": {"name":"infra-aws-core-2-default-rts7j","namespace":"infra"}, "name": "infra-aws-core-2-default-rts7j-runner-77qj7", "workflowRunId": 42047428183, "jobId": "3fa9d8e3-1d97-5989-9d6e-e4038410b5ba"}
2026-11-05T14:43:57Z	INFO	EphemeralRunnerSet	Skipping ephemeral runner since it is running a job	{"version": "0.15.1", "ephemeralrunnerset": {"name":"infra-aws-core-2-default-rts7j","namespace":"infra"}, "name": "infra-aws-core-2-default-rts7j-runner-77qj7", "workflowRunId": 42047428183, "jobId": "3fa9d8e3-1d97-5989-9d6e-e4038410b5ba"}
2026-11-05T14:43:57Z	INFO	EphemeralRunnerSet	Skipping ephemeral runner since it is running a job	{"version": "0.15.1", "ephemeralrunnerset": {"name":"infra-aws-core-2-default-rts7j","namespace":"infra"}, "name": "infra-aws-core-2-default-rts7j-runner-77qj7", "workflowRunId": 42047428183, "jobId": "3fa9d8e3-1d97-5989-9d6e-e4038410b5ba"}
Runner Pod Logs
Run 3: job log of the cancelled job
2026-11-05T14:43:49.9103757Z Current runner version: '2.338.1'
2026-11-05T14:43:49.9114558Z Runner name: 'infra-aws-core-2-default-rts7j-runner-8pjml'
2026-11-05T14:43:49.9116421Z Runner group name: 'infra-dev-self-hosted-runners'
2026-11-05T14:43:49.9118353Z Machine name: 'infra-aws-core-2-default-rts7j-runner-8pjml'
2026-11-05T14:43:49.9124877Z ##[group]GITHUB_TOKEN Permissions
2026-11-05T14:43:49.9128855Z Metadata: read
2026-11-05T14:43:49.9129771Z ##[endgroup]
2026-11-05T14:43:49.9133335Z Secret source: Actions
2026-11-05T14:43:49.9134775Z Cache mode: write
2026-11-05T14:43:49.9136019Z Prepare workflow directory
2026-11-05T14:43:49.9691728Z Prepare all required actions
2026-11-05T14:43:49.9867996Z Complete job name: test (dev-aws-core-2-default)
2026-11-05T14:43:50.1409078Z A job started hook has been configured by the self-hosted runner administrator
2026-11-05T14:43:50.1669201Z ##[group]Run '/home/runner/hooks/label-runner-pod.sh'
2026-11-05T14:43:50.1689061Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2026-11-05T14:43:50.1691119Z ##[endgroup]
2026-11-05T14:43:50.6453779Z Client Version: v1.35.8
2026-11-05T14:43:50.6455320Z Kustomize Version: v5.7.1
2026-11-05T14:43:50.6456952Z Server Version: v1.36.4-eks-cfb47f5
2026-11-05T14:43:50.6459249Z Adding label 'camunda.cloud/workflow=Test-dev-self-hosted-runners' to the runner
2026-11-05T14:43:50.6647463Z Adding label 'camunda.cloud/workflow-repository=camunda-infra-core' to the runner
2026-11-05T14:43:50.8239381Z Adding label 'camunda.cloud/job-id=test' to the runner
2026-11-05T14:43:50.9302870Z Adding label 'camunda.cloud/run-id=42047639016' to the runner
2026-11-05T14:43:51.0760490Z Adding label 'camunda.cloud/run-attempt=1' to the runner
2026-11-05T14:43:51.1591572Z Adding label 'camunda.cloud/github-ref=heads-stage' to the runner
2026-11-05T14:43:51.3887508Z Adding label 'camunda.cloud/base-ref=not-applicable' to the runner
2026-11-05T14:43:51.5314290Z ##[group]Run # The runner name tells which fleet picked the job up (maint-* for the maintenance sets).
2026-11-05T14:43:51.5317635Z # The runner name tells which fleet picked the job up (maint-* for the maintenance sets).
2026-11-05T14:43:51.5320322Z echo "Running on ${RUNNER_NAME} (label ${LABEL}) at $(date -u +%FT%TZ)"
2026-11-05T14:43:51.5323417Z sleep "$SLEEP_SECONDS"
2026-11-05T14:43:51.5324700Z echo "Finished at $(date -u +%FT%TZ)"
2026-11-05T14:43:51.5339535Z shell: /usr/bin/bash -e {0}
2026-11-05T14:43:51.5340790Z env:
2026-11-05T14:43:51.5341730Z   SLEEP_SECONDS: 900
2026-11-05T14:43:51.5342831Z   LABEL: dev-aws-core-2-default
2026-11-05T14:43:51.5343908Z ##[endgroup]
2026-11-05T14:43:51.5419819Z Running on infra-aws-core-2-default-rts7j-runner-8pjml (label dev-aws-core-2-default) at 2026-11-05T14:43:51Z
2026-11-05T14:43:57.1238858Z ##[error]Process completed with exit code 130.
2026-11-05T14:43:57.1424296Z Cleaning up orphan processes
2026-11-05T14:43:57.1555198Z ##[error]The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.
Dominant language
Go
Stars
6.7k
Forks
1.5k
Avg merge
1h 34m
Merged PRs (30d)
5

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from actions/actions-runner-controller

All issues in actions/actions-runner-controller

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.