Runner cancels a healthy running job after 10 cumulative (not consecutive) retriable broker poll errors

未关闭 适合新手
#4,820 3 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 1 天内回复

还没有人认领这个 Issue。

评估

难度
2/5
预计耗时
1-3 小时
新手友好度
76/100
Issue 类型
缺陷
描述清晰度
描述清楚
活跃度
活跃
技术栈
csharp
领域
backend

调研方向

从 src/Runner.Listener/BrokerMessageListener.cs 中的 GetNextMessageAsync 开始,跟踪轮询成功或抛出异常时 continuousError 和 encounteringError 如何变化。Issue 将无消息的轮询路径认定为一次成功的轮询,应重置计数器。请使用相关的现有测试(如果有)确认这一行为,并检查只有连续失败的轮询才会达到重试上限。

由索引模型根据 Issue 内容生成。

描述

Version / environment: runner v2.338.0 (the same code is on main; I did not read the v2.338.1 tag itself), Linux ARM64, self-hosted, systemd service.

What happens

While a job is running, the listener keeps polling the broker (GET .../message?...&status=Busy). On our network some of these long-polls hit the client's 100 s HTTP timeout. Each one that escapes the inner request retry is logged as Retriable exception: The HTTP request timed out after 00:01:40. and the listener backs off and polls again, as intended.

But after the 10th such exception during the same job, the listener shuts down its job dispatcher and sends a cancellation to the worker, even though the job itself is healthy (job renewals were still succeeding). The job ends as cancelled. On our machine a retriable exception escapes about every 9.5 minutes, so every job that runs longer than about 95 minutes is cancelled at that point. Three jobs in one day were cancelled at 95.4, 95.6 and 96.1 minutes; a job that finished at 93.75 minutes passed.

Log sequence at the cancellation (session ids removed):

[... INFO BrokerMessageListener] Retriable exception: The HTTP request timed out after 00:01:40.
[... INFO JobDispatcher] Shutting down JobDispatcher. Make sure all WorkerDispatcher has finished.
[... INFO JobDispatcher] Ensure WorkerDispatcher for job ... run to finish, cancel any running job.
[... INFO JobDispatcher] Send job cancellation message to worker for job ...
[... INFO JobDispatcher] finish job request for job ... with result: Canceled

In both logs I checked, this was exactly the 10th Retriable exception after the job started.

Cause in the source

src/Runner.Listener/BrokerMessageListener.cs, GetNextMessageAsync (v2.338.0, lines 281-371; the same lines on main):

  • line 281: int continuousError = 0;
  • line 284: var maxRetryCount = 10;
  • line 361: continuousError++; on every retriable exception
  • lines 369-371: else if (continuousError >= maxRetryCount) { throw; }

continuousError is never set back to 0. A poll that returns no message just does continue (lines ~306-309), and nothing else resets it. The call to GetNextMessageAsync stays open for the whole time a job runs, so the counter accumulates every poll failure over the job's lifetime, not consecutive ones as the name and the "random backoff ... consecutive error" comments suggest. Any sufficiently long job on a network with occasional dropped long-polls will eventually reach 10.

Expected behaviour

Only 10 consecutive failed polls should give up. A successful poll (including one that returns no message) should reset the counter.

Possible fix

Reset continuousError (and encounteringError) to 0 after a poll completes without throwing, i.e. just before the if (message == null) { continue; } check. I have not built or tested a patched runner. It may also be worth making maxRetryCount configurable.

Workaround

Keep jobs shorter than the point where the 10th escaped timeout happens (about 95 minutes for us).

Not established

Why the broker long-polls time out at 100 s on our network. They also happen on idle runners, and a renewjob POST sometimes fails with "The response ended prematurely". That may be our network; this issue is only about the listener treating cumulative errors as consecutive.

主要语言
C#
星标
6.4k
派生
1.5k
平均合并
1 天 7 小时
30 天内合并 PR
36

环境准备

在 Codespaces 中打开

在浏览器里用你自己的 GitHub 账号启动这个项目的开发容器。

  • 没有 Dockerfile 或 Docker Compose 文件
  • 没有 Pull Request 模板
  • 没有贡献指南

从这里开始

  1. 先读完整个 Issue,再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库,在一个分支上完成修改。
  4. 提交 Pull Request,并在描述里引用这个 Issue 编号。

actions/runner 的其他 Issue

查看 actions/runner 的全部 Issue

相似的 Issue

更多 C# Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。