Runner cancels a healthy running job after 10 cumulative (not consecutive) retriable broker poll errors

Abierto Apto para principiantes
#4,820 3 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
2/5
Tiempo estimado
1-3 horas
Aptitud para principiantes
76/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
csharp
Área
backend

Línea de trabajo

Empieza en src/Runner.Listener/BrokerMessageListener.cs, en GetNextMessageAsync, y sigue cómo cambian continuousError y encounteringError cuando un sondeo tiene éxito o lanza una excepción. El issue identifica la ruta de sondeo sin mensajes como un sondeo correcto que debería reiniciar los contadores. Confirma el comportamiento con las pruebas existentes pertinentes, si las hay, y comprueba que solo los sondeos fallidos consecutivos alcanzan el límite de reintentos.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Version / environment: runner v2.338.0 (the same code is on main; I did not read the v2.338.1 tag itself), Linux ARM64, self-hosted, systemd service.

What happens

While a job is running, the listener keeps polling the broker (GET .../message?...&status=Busy). On our network some of these long-polls hit the client's 100 s HTTP timeout. Each one that escapes the inner request retry is logged as Retriable exception: The HTTP request timed out after 00:01:40. and the listener backs off and polls again, as intended.

But after the 10th such exception during the same job, the listener shuts down its job dispatcher and sends a cancellation to the worker, even though the job itself is healthy (job renewals were still succeeding). The job ends as cancelled. On our machine a retriable exception escapes about every 9.5 minutes, so every job that runs longer than about 95 minutes is cancelled at that point. Three jobs in one day were cancelled at 95.4, 95.6 and 96.1 minutes; a job that finished at 93.75 minutes passed.

Log sequence at the cancellation (session ids removed):

[... INFO BrokerMessageListener] Retriable exception: The HTTP request timed out after 00:01:40.
[... INFO JobDispatcher] Shutting down JobDispatcher. Make sure all WorkerDispatcher has finished.
[... INFO JobDispatcher] Ensure WorkerDispatcher for job ... run to finish, cancel any running job.
[... INFO JobDispatcher] Send job cancellation message to worker for job ...
[... INFO JobDispatcher] finish job request for job ... with result: Canceled

In both logs I checked, this was exactly the 10th Retriable exception after the job started.

Cause in the source

src/Runner.Listener/BrokerMessageListener.cs, GetNextMessageAsync (v2.338.0, lines 281-371; the same lines on main):

  • line 281: int continuousError = 0;
  • line 284: var maxRetryCount = 10;
  • line 361: continuousError++; on every retriable exception
  • lines 369-371: else if (continuousError >= maxRetryCount) { throw; }

continuousError is never set back to 0. A poll that returns no message just does continue (lines ~306-309), and nothing else resets it. The call to GetNextMessageAsync stays open for the whole time a job runs, so the counter accumulates every poll failure over the job's lifetime, not consecutive ones as the name and the "random backoff ... consecutive error" comments suggest. Any sufficiently long job on a network with occasional dropped long-polls will eventually reach 10.

Expected behaviour

Only 10 consecutive failed polls should give up. A successful poll (including one that returns no message) should reset the counter.

Possible fix

Reset continuousError (and encounteringError) to 0 after a poll completes without throwing, i.e. just before the if (message == null) { continue; } check. I have not built or tested a patched runner. It may also be worth making maxRetryCount configurable.

Workaround

Keep jobs shorter than the point where the 10th escaped timeout happens (about 95 minutes for us).

Not established

Why the broker long-polls time out at 100 s on our network. They also happen on idle runners, and a renewjob POST sometimes fails with "The response ended prematurely". That may be our network; this issue is only about the listener treating cumulative errors as consecutive.

Lenguaje dominante
C#
Estrellas
6.4k
Forks
1.5k
Merge medio
1 d 7 h
PR fusionados (30 d)
36

Preparar el entorno

Abrir en Codespaces

Inicia el contenedor de desarrollo del proyecto en tu navegador, con tu propia cuenta de GitHub.

  • Sin Dockerfile ni archivo de Docker Compose
  • Sin plantilla de pull request
  • Sin guía de contribución

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de actions/runner

Todos los issues de actions/runner

Issues similares

Más issues de C#

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.