Runner cancels a healthy running job after 10 cumulative (not consecutive) retriable broker poll errors
Los mantenedores suelen responder en 1 día
Nadie ha tomado este issue todavía.
Evaluación
- Dificultad
- 2/5
- Tiempo estimado
- 1-3 horas
- Aptitud para principiantes
- 76/100
Línea de trabajo
Empieza en src/Runner.Listener/BrokerMessageListener.cs, en GetNextMessageAsync, y sigue cómo cambian continuousError y encounteringError cuando un sondeo tiene éxito o lanza una excepción. El issue identifica la ruta de sondeo sin mensajes como un sondeo correcto que debería reiniciar los contadores. Confirma el comportamiento con las pruebas existentes pertinentes, si las hay, y comprueba que solo los sondeos fallidos consecutivos alcanzan el límite de reintentos.
Escrito por el modelo de indexación a partir del texto del issue.
Descripción
Version / environment: runner v2.338.0 (the same code is on main; I did not read the v2.338.1 tag itself), Linux ARM64, self-hosted, systemd service.
What happens
While a job is running, the listener keeps polling the broker (GET .../message?...&status=Busy). On our network some of these long-polls hit the client's 100 s HTTP timeout. Each one that escapes the inner request retry is logged as Retriable exception: The HTTP request timed out after 00:01:40. and the listener backs off and polls again, as intended.
But after the 10th such exception during the same job, the listener shuts down its job dispatcher and sends a cancellation to the worker, even though the job itself is healthy (job renewals were still succeeding). The job ends as cancelled. On our machine a retriable exception escapes about every 9.5 minutes, so every job that runs longer than about 95 minutes is cancelled at that point. Three jobs in one day were cancelled at 95.4, 95.6 and 96.1 minutes; a job that finished at 93.75 minutes passed.
Log sequence at the cancellation (session ids removed):
[... INFO BrokerMessageListener] Retriable exception: The HTTP request timed out after 00:01:40.
[... INFO JobDispatcher] Shutting down JobDispatcher. Make sure all WorkerDispatcher has finished.
[... INFO JobDispatcher] Ensure WorkerDispatcher for job ... run to finish, cancel any running job.
[... INFO JobDispatcher] Send job cancellation message to worker for job ...
[... INFO JobDispatcher] finish job request for job ... with result: Canceled
In both logs I checked, this was exactly the 10th Retriable exception after the job started.
Cause in the source
src/Runner.Listener/BrokerMessageListener.cs, GetNextMessageAsync (v2.338.0, lines 281-371; the same lines on main):
- line 281:
int continuousError = 0; - line 284:
var maxRetryCount = 10; - line 361:
continuousError++;on every retriable exception - lines 369-371:
else if (continuousError >= maxRetryCount) { throw; }
continuousError is never set back to 0. A poll that returns no message just does continue (lines ~306-309), and nothing else resets it. The call to GetNextMessageAsync stays open for the whole time a job runs, so the counter accumulates every poll failure over the job's lifetime, not consecutive ones as the name and the "random backoff ... consecutive error" comments suggest. Any sufficiently long job on a network with occasional dropped long-polls will eventually reach 10.
Expected behaviour
Only 10 consecutive failed polls should give up. A successful poll (including one that returns no message) should reset the counter.
Possible fix
Reset continuousError (and encounteringError) to 0 after a poll completes without throwing, i.e. just before the if (message == null) { continue; } check. I have not built or tested a patched runner. It may also be worth making maxRetryCount configurable.
Workaround
Keep jobs shorter than the point where the 10th escaped timeout happens (about 95 minutes for us).
Not established
Why the broker long-polls time out at 100 s on our network. They also happen on idle runners, and a renewjob POST sometimes fails with "The response ended prematurely". That may be our network; this issue is only about the listener treating cumulative errors as consecutive.
- Lenguaje dominante
- C#
- Estrellas
- 6.4k
- Forks
- 1.5k
- Merge medio
- 1 d 7 h
- PR fusionados (30 d)
- 36
Preparar el entorno
Inicia el contenedor de desarrollo del proyecto en tu navegador, con tu propia cuenta de GitHub.
- Sin Dockerfile ni archivo de Docker Compose
- Sin plantilla de pull request
- Sin guía de contribución
Primeros pasos
- Lee el issue completo y luego la guía de contribución del proyecto.
- Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
- Haz un fork del repositorio y trabaja en una rama.
- Abre un pull request que haga referencia al número del issue.
Más de actions/runner
-
Workflow schema omits documented code-quality and copilot-requests permissionsPosiblemente ocupada @localzet la tomó hace 12 días. Abierto
Dificultad 2/5 1-3 horas Aptitud para principiantes 78/100
Los mantenedores suelen responder en 1 día
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 78/100
actions/runner#4692 · 1 comentario ·
Los mantenedores suelen responder en 1 día
-
Runner crashes at InitializeSecretMasker on machines with Thai (th-TH) culture — culture-sensitive LastIndexOf in PowerShellPreAmpersandEscapePosiblemente ocupada @CloudNinjaDev la tomó hace 47 días. Abierto
Dificultad 2/5 1-3 horas Aptitud para principiantes 88/100
Los mantenedores suelen responder en 1 día
-
Broker retry logs an ERR stack trace and a phantom backoff on every job completion (listener's own cancellation treated as a request failure)Posiblemente ocupada @nitrobass24 la tomó hace 87 días. Abierto
Dificultad 2/5 1-3 horas Aptitud para principiantes 78/100
Los mantenedores suelen responder en 1 día
-
bug
Dificultad 2/5 1-3 horas Aptitud para principiantes 72/100
actions/runner#4600 · 1 comentario ·
Los mantenedores suelen responder en 1 día
Todos los issues de actions/runner
Issues similares
-
needs triage
Dificultad 1/5 Menos de una hora Aptitud para principiantes 71/100
Los mantenedores suelen responder en 1 día
-
bug
Dificultad 2/5 1-3 horas Aptitud para principiantes 66/100
majorsilence/Majorsilence.Forms#718 ·
Los mantenedores suelen responder en 1 día
-
bug debugger flaky small
Dificultad 2/5 1-3 horas Aptitud para principiantes 74/100
Los mantenedores suelen responder en 1 día
-
[BUG] MyAnonaMouse search turns a curly apostrophe into a space, so titles with ’ are never foundAbierto
Dificultad 2/5 1-3 horas Aptitud para principiantes 74/100
Los mantenedores suelen responder en 1 día
-
documentation
Dificultad 1/5 Menos de una hora Aptitud para principiantes 90/100
Los mantenedores suelen responder en 1 día