Queued jobs remain stuck after restoring maxRunners from 0 to 3; dispatching a new workflow wakes them
I maintainer di solito rispondono entro 1 giorno
Nessuno ha ancora preso questa issue.
Valutazione
- Difficoltà
- 4/5
- Tempo stimato
- 1-2 giorni
- Idoneità per principianti
- 15/100
Direzione di ricerca
Start with the reproduction repository at https://github.com/kuring/arc-test and the listener log line "assigned job"=0 decision=0 min=0 max=3 that the report describes, which shows the listener sees no jobs to assign after capacity is restored. Compare that with the EphemeralRunnerSet replica behaviour the report describes. Done means jobs queued before a 0-to-3 restore get runners without a new dispatch, and this is confirmed on a kind cluster with the same Helm charts.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Descrizione
Checks
- I've already read https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/troubleshooting-actions-runner-controller-errors and I'm sure my issue is not covered in the troubleshooting guide.
- I am using charts that are officially provided
Controller Version
0.15.0
Deployment Method
Helm
Checks
- This isn't a question or user support case (For Q&A and community support, go to Discussions).
- I've read the Changelog before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes
To Reproduce
1. Install the official `gha-runner-scale-set-controller` and `gha-runner-scale-set` Helm charts, both at version `0.15.0`. Use the official runner image `ghcr.io/actions/actions-runner:2.338.0` with `minRunners: 0` and `maxRunners: 3`. Verify that a baseline workflow completes successfully on this scale set.1. Install the official gha-runner-scale-set-controller and gha-runner-scale-set Helm charts, both at version 0.15.0. Use ghcr.io/actions/actions-runner:2.338.0 with minRunners=0 and maxRunners=3. Verify that a baseline workflow completes successfully.
2. Update the scale set through Helm to minRunners=0,maxRunners=0. Wait until the capacity configuration has converged and no active runner pods remain.
3. Dispatch a workflow containing three independent jobs targeting this exact scale set. Each job only sleeps for 30 seconds.
4. Confirm that all three jobs remain queued, without assigned runners, for at least 60 seconds.
5. Restore minRunners=0,maxRunners=3 through the same official Helm chart. Confirm that both the AutoscalingRunnerSet and listener report maximum capacity 3, and that the listener is Ready.
6. Observe the original jobs for at least 600 seconds without submitting other jobs or performing additional interventions. The jobs remain queued and no runner pods are created.
7. As a separate diagnostic step, dispatch one additional job to the same scale set.
8. The original three jobs then receive runners and complete successfully with their original job IDs.
Reproduction repository:
https://github.com/kuring/arc-test
The exact Helm commands and configuration are included in the reproduction repository.
Describe the bug
With official ARC 0.15.0 and runner 2.338.0, jobs submitted while minRunners=0,maxRunners=0 remain queued after restoring minRunners=0,maxRunners=3. Capacity-only recovery stalls, but several additional interventions have allowed the same queued jobs to execute.
Affected scenario
Normal-capacity baseline jobs execute successfully. The failure occurs after this sequence:
- Set both runner limits to 0 and wait until no active runner pods remain.
- Submit jobs while capacity is 0; they remain queued without assigned runners.
- Restore
maxRunners=3, keepingminRunners=0.
Although both the AutoscalingRunnerSet and listener report maximum capacity 3 and the listener is Ready, the original jobs remain queued and no runner pods are created. In the 2026-10-15 reproduction, this persisted for approximately 605 seconds (108 samples spanning 604.846 seconds). The listener repeatedly logged "assigned job"=0 decision=0 min=0 max=3.
Operations that recovered the existing queue, and operations that did not
Each intervention below was tested separately against an existing stalled queue after capacity had been restored. These are observations from specific experiments, not guaranteed workarounds for every environment.
| Operation after capacity is restored | Observed result |
|---|---|
Keep minRunners=0,maxRunners=3 and wait, with no additional intervention |
Original jobs remained queued without runners for more than 600 seconds. |
Dispatch one additional job to the same scale set, keeping minRunners=0,maxRunners=3 |
Original queued jobs received runners and completed successfully with unchanged job IDs. No additional listener restart or manual runner creation was needed. |
Increase minRunners from 0 to 1 while keeping maxRunners=3 |
Original queue recovered in a separate manual observation. No dedicated evidence directory was retained for this case, so this result has weaker supporting evidence. |
Temporarily patch the existing EphemeralRunnerSet's spec.replicas from 0 to 1, keeping minRunners=0,maxRunners=3 |
In a separate diagnostic experiment, the original queued jobs completed with unchanged IDs, without dispatching a new workflow or restarting the listener. The listener subsequently scaled the set normally and returned it to 0 after completion. This was a low-level diagnostic intervention, not a tested operational workaround. |
Restart only the listener while keeping minRunners=0,maxRunners=3 |
In the full rerun, the replacement listener became Ready, but the original jobs remained queued for the 600-second recovery window. They completed only after an additional diagnostic workflow was dispatched. Complete logs and pod descriptions are included in the attached Gists. |
The minRunners=1 observation concerns changing the minimum after the queue was already stalled. It does not establish that keeping minRunners=1 throughout other pause/resume sequences would always prevent this problem.
Evidence and interpretation
The attached Gists contain a full rerun on 2026-10-15, from 07:07 to 07:51 UTC. Smoke passed; startup, resume, and restart each timed out during the 600-second recovery wait. In all three cases, the original queued jobs subsequently completed successfully after a new diagnostic workflow was dispatched. The original recovery results remain failed. The report in each Gist maps scenarios to their original and diagnostic workflows.
In the earlier new-job experiment on 2026-10-15, capacity was restored at 04:12:14 UTC. The original jobs stayed queued through 04:22:19. An additional workflow was dispatched at 04:22:26; the listener reported "assigned job"=3 decision=3 at 04:22:29.319, runner pods were created at 04:22:31, and the original jobs started at 04:22:38 and completed successfully at 04:23:14.
Original workflow: https://github.com/kuring/arc-test/actions/runs/38680120162
Original job IDs: 116242511614, 116242511833, 116242511848.
Additional workflow: https://github.com/kuring/arc-test/actions/runs/38681115753
Separate temporary-runner experiment: https://github.com/kuring/arc-test/actions/runs/38538867417
These workflow links now show successful completion because of the later interventions; they do not mean that restoring capacity alone succeeded. The original jobs were not canceled or re-run.
The Helm capacity updates recreated the listener. In the new-job experiment, the replacement listener UID remained unchanged from capacity restoration through the stalled period and recovery. Assignment messages arrived before the runner pods were created, so the successful interventions cannot all be attributed to runner session creation. The available evidence does not determine whether the underlying cause is in ARC or the GitHub Actions service.
Describe the expected behavior
Once minRunners=0,maxRunners=3 has converged and the listener is Ready, jobs already queued for the scale set should be scheduled automatically, up to the configured limit, and complete without requiring a new workflow dispatch, an extra listener restart, or an increase in minRunners.
Additional Context
environment:
controller_chart: gha-runner-scale-set-controller
controller_chart_version: "0.15.0"
scale_set_chart: gha-runner-scale-set
scale_set_chart_version: "0.15.0"
runner_image: ghcr.io/actions/actions-runner:2.338.0
kubernetes: v1.33.1
kind: v0.29.0
helm: v3.14.4
architecture: amd64
scope: repository
custom_proxy: false
docker_in_docker: false
extra_scheduling_components: false
controller_values: {}
scale_set_values:
githubConfigUrl: https://github.com/kuring/arc-test
githubConfigSecret: arc-auth # Existing Secret; credentials omitted.
runnerScaleSetName: arc-test-79ffd92df47f
minRunners: 0
maxRunners: 3
template:
spec:
containers:
- name: runner
image: ghcr.io/actions/actions-runner:2.338.0
command: ["/home/runner/run.sh"]
capacity_sequence:
- {minRunners: 0, maxRunners: 3}
- {minRunners: 0, maxRunners: 0}
- {minRunners: 0, maxRunners: 3}
Controller Logs
https://gist.github.com/kuring/9a9ab403bdadeb1e5d481a1746f9ed4f
Runner Pod Logs
https://gist.github.com/kuring/f52121f38a2f5bb0d89285e65b41879b
- Lingua principale
- Go
- Stelle
- 6.7k
- Fork
- 1.5k
- Merge medio
- 1h 34m
- PR unite (30g)
- 5
Preparare l'ambiente
- Include un Dockerfile o un file Docker Compose
- Nessun modello di pull request
- Leggi la guida per i contributori
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Altre issue di actions/actions-runner-controller
-
community enhancement needs triage
Difficoltà 2/5 1-3 ore Idoneità per principianti 68/100
actions/actions-runner-controller#3997 · 9 commenti · 27 reazioni ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 4/5 3-5 giorni Idoneità per principianti 48/100
actions/actions-runner-controller#4739 · 1 commento · 4 reazioni ·
I maintainer di solito rispondono entro 1 giorno
-
gha-runner-scale-set: support Google Cloud Secret Manager as a vault for GitHub credentialsForse già presa Una pull request collegata a questa issue è aperta o già unita. Apertacommunity enhancement needs triage
Difficoltà 4/5 3-5 giorni Idoneità per principianti 52/100
actions/actions-runner-controller#4737 · 1 commento ·
I maintainer di solito rispondono entro 1 giorno
-
Controller upgrade cancels jobs that started shortly before their runner set is deletedForse di nuovo libera @ankit373 l’ha presa 9 giorni fa e non c’è nessuna pull request aperta. Aperta
Difficoltà 4/5 3-5 giorni Idoneità per principianti 45/100
actions/actions-runner-controller#4736 · 2 commenti ·
I maintainer di solito rispondono entro 1 giorno
-
EphemeralRunnerSet controller creates more runners than `spec.replicas` when its cache lags its own createsForse già presa @jia-pylon l’ha presa 14 giorni fa. Aperta
Difficoltà 4/5 3-5 giorni Idoneità per principianti 30/100
actions/actions-runner-controller#4734 · 1 commento ·
I maintainer di solito rispondono entro 1 giorno
Tutte le issue di actions/actions-runner-controller
Issue simili
-
benchmark
Difficoltà 1/5 1-3 ore Idoneità per principianti 72/100
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 77/100
mozilla-ai/cq#601 ·
I maintainer di solito rispondono entro 1 giorno
-
documentation migration
Difficoltà 2/5 1-3 ore Idoneità per principianti 75/100
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 84/100
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 78/100
I maintainer di solito rispondono entro 1 giorno