Queued jobs remain stuck after restoring maxRunners from 0 to 3; dispatching a new workflow wakes them

Aperta
#4,715 2 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
1-2 giorni
Idoneità per principianti
15/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
github-actions, go, helm, kubernetes
Ambito
devops

Direzione di ricerca

Start with the reproduction repository at https://github.com/kuring/arc-test and the listener log line "assigned job"=0 decision=0 min=0 max=3 that the report describes, which shows the listener sees no jobs to assign after capacity is restored. Compare that with the EphemeralRunnerSet replica behaviour the report describes. Done means jobs queued before a 0-to-3 restore get runners without a new dispatch, and this is confirmed on a kind cluster with the same Helm charts.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

bug gha-runner-scale-set needs triage
Checks
Controller Version

0.15.0

Deployment Method

Helm

Checks
  • This isn't a question or user support case (For Q&A and community support, go to Discussions).
  • I've read the Changelog before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes
To Reproduce
1. Install the official `gha-runner-scale-set-controller` and `gha-runner-scale-set` Helm charts, both at version `0.15.0`. Use the official runner image `ghcr.io/actions/actions-runner:2.338.0` with `minRunners: 0` and `maxRunners: 3`. Verify that a baseline workflow completes successfully on this scale set.1. Install the official gha-runner-scale-set-controller and gha-runner-scale-set Helm charts, both at version 0.15.0. Use ghcr.io/actions/actions-runner:2.338.0 with minRunners=0 and maxRunners=3. Verify that a baseline workflow completes successfully.

2. Update the scale set through Helm to minRunners=0,maxRunners=0. Wait until the capacity configuration has converged and no active runner pods remain.

3. Dispatch a workflow containing three independent jobs targeting this exact scale set. Each job only sleeps for 30 seconds.

4. Confirm that all three jobs remain queued, without assigned runners, for at least 60 seconds.

5. Restore minRunners=0,maxRunners=3 through the same official Helm chart. Confirm that both the AutoscalingRunnerSet and listener report maximum capacity 3, and that the listener is Ready.

6. Observe the original jobs for at least 600 seconds without submitting other jobs or performing additional interventions. The jobs remain queued and no runner pods are created.

7. As a separate diagnostic step, dispatch one additional job to the same scale set.

8. The original three jobs then receive runners and complete successfully with their original job IDs.

Reproduction repository:
https://github.com/kuring/arc-test

The exact Helm commands and configuration are included in the reproduction repository.
Describe the bug

With official ARC 0.15.0 and runner 2.338.0, jobs submitted while minRunners=0,maxRunners=0 remain queued after restoring minRunners=0,maxRunners=3. Capacity-only recovery stalls, but several additional interventions have allowed the same queued jobs to execute.

Affected scenario

Normal-capacity baseline jobs execute successfully. The failure occurs after this sequence:

  1. Set both runner limits to 0 and wait until no active runner pods remain.
  2. Submit jobs while capacity is 0; they remain queued without assigned runners.
  3. Restore maxRunners=3, keeping minRunners=0.

Although both the AutoscalingRunnerSet and listener report maximum capacity 3 and the listener is Ready, the original jobs remain queued and no runner pods are created. In the 2026-10-15 reproduction, this persisted for approximately 605 seconds (108 samples spanning 604.846 seconds). The listener repeatedly logged "assigned job"=0 decision=0 min=0 max=3.

Operations that recovered the existing queue, and operations that did not

Each intervention below was tested separately against an existing stalled queue after capacity had been restored. These are observations from specific experiments, not guaranteed workarounds for every environment.

Operation after capacity is restored Observed result
Keep minRunners=0,maxRunners=3 and wait, with no additional intervention Original jobs remained queued without runners for more than 600 seconds.
Dispatch one additional job to the same scale set, keeping minRunners=0,maxRunners=3 Original queued jobs received runners and completed successfully with unchanged job IDs. No additional listener restart or manual runner creation was needed.
Increase minRunners from 0 to 1 while keeping maxRunners=3 Original queue recovered in a separate manual observation. No dedicated evidence directory was retained for this case, so this result has weaker supporting evidence.
Temporarily patch the existing EphemeralRunnerSet's spec.replicas from 0 to 1, keeping minRunners=0,maxRunners=3 In a separate diagnostic experiment, the original queued jobs completed with unchanged IDs, without dispatching a new workflow or restarting the listener. The listener subsequently scaled the set normally and returned it to 0 after completion. This was a low-level diagnostic intervention, not a tested operational workaround.
Restart only the listener while keeping minRunners=0,maxRunners=3 In the full rerun, the replacement listener became Ready, but the original jobs remained queued for the 600-second recovery window. They completed only after an additional diagnostic workflow was dispatched. Complete logs and pod descriptions are included in the attached Gists.

The minRunners=1 observation concerns changing the minimum after the queue was already stalled. It does not establish that keeping minRunners=1 throughout other pause/resume sequences would always prevent this problem.

Evidence and interpretation

The attached Gists contain a full rerun on 2026-10-15, from 07:07 to 07:51 UTC. Smoke passed; startup, resume, and restart each timed out during the 600-second recovery wait. In all three cases, the original queued jobs subsequently completed successfully after a new diagnostic workflow was dispatched. The original recovery results remain failed. The report in each Gist maps scenarios to their original and diagnostic workflows.

Scenario Original queued workflow Diagnostic workflow
startup https://github.com/kuring/arc-test/actions/runs/38696819359 https://github.com/kuring/arc-test/actions/runs/38698070634
resume https://github.com/kuring/arc-test/actions/runs/38698289447 https://github.com/kuring/arc-test/actions/runs/38699906732
restart https://github.com/kuring/arc-test/actions/runs/38700090077 https://github.com/kuring/arc-test/actions/runs/38701394963

In the earlier new-job experiment on 2026-10-15, capacity was restored at 04:12:14 UTC. The original jobs stayed queued through 04:22:19. An additional workflow was dispatched at 04:22:26; the listener reported "assigned job"=3 decision=3 at 04:22:29.319, runner pods were created at 04:22:31, and the original jobs started at 04:22:38 and completed successfully at 04:23:14.

Original workflow: https://github.com/kuring/arc-test/actions/runs/38680120162
Original job IDs: 116242511614, 116242511833, 116242511848.
Additional workflow: https://github.com/kuring/arc-test/actions/runs/38681115753
Separate temporary-runner experiment: https://github.com/kuring/arc-test/actions/runs/38538867417

These workflow links now show successful completion because of the later interventions; they do not mean that restoring capacity alone succeeded. The original jobs were not canceled or re-run.

The Helm capacity updates recreated the listener. In the new-job experiment, the replacement listener UID remained unchanged from capacity restoration through the stalled period and recovery. Assignment messages arrived before the runner pods were created, so the successful interventions cannot all be attributed to runner session creation. The available evidence does not determine whether the underlying cause is in ARC or the GitHub Actions service.

Describe the expected behavior

Once minRunners=0,maxRunners=3 has converged and the listener is Ready, jobs already queued for the scale set should be scheduled automatically, up to the configured limit, and complete without requiring a new workflow dispatch, an extra listener restart, or an increase in minRunners.

Additional Context
environment:
  controller_chart: gha-runner-scale-set-controller
  controller_chart_version: "0.15.0"
  scale_set_chart: gha-runner-scale-set
  scale_set_chart_version: "0.15.0"
  runner_image: ghcr.io/actions/actions-runner:2.338.0
  kubernetes: v1.33.1
  kind: v0.29.0
  helm: v3.14.4
  architecture: amd64
  scope: repository
  custom_proxy: false
  docker_in_docker: false
  extra_scheduling_components: false

controller_values: {}

scale_set_values:
  githubConfigUrl: https://github.com/kuring/arc-test
  githubConfigSecret: arc-auth  # Existing Secret; credentials omitted.
  runnerScaleSetName: arc-test-79ffd92df47f
  minRunners: 0
  maxRunners: 3
  template:
    spec:
      containers:
        - name: runner
          image: ghcr.io/actions/actions-runner:2.338.0
          command: ["/home/runner/run.sh"]

capacity_sequence:
  - {minRunners: 0, maxRunners: 3}
  - {minRunners: 0, maxRunners: 0}
  - {minRunners: 0, maxRunners: 3}
Controller Logs
https://gist.github.com/kuring/9a9ab403bdadeb1e5d481a1746f9ed4f
Runner Pod Logs
https://gist.github.com/kuring/f52121f38a2f5bb0d89285e65b41879b
Lingua principale
Go
Stelle
6.7k
Fork
1.5k
Merge medio
1h 34m
PR unite (30g)
5

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di actions/actions-runner-controller

Tutte le issue di actions/actions-runner-controller

Issue simili

Altre issue su Go

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.