Branch build clones: mirror stalls, clone timeout and controller memory
维护者通常 2 天内回复
@bryancall 已经在做这个了。
开始于 2026年10月14日。
评估
这个 Issue 还没有评估数据。
描述
Branch builds keep failing in the Clone stage, and the controller keeps running out of memory. I dug into it on Oct 13 and 14, and there are three separate problems:
- ATS reused connections to the mirror after the mirror had closed them. Each reuse sat for 180 s before ATS retried. On Oct 13 there were 30 of these stalls on the mirror, 23 on the docs site and 8 on Jenkins.
- Branch builds fetch every branch at depth 1000. One fetch takes about 15 s and about 125 MB of private memory on the controller. A branch build starts about 30 children at once on a 4 CPU, 8 GB box with no swap, so the fetches overlap and it runs out of memory. At the first Jenkins kill on Oct 14 (12:43 PDT), the kernel's out-of-memory report had 21 fetches in flight using 3,317 MB of git memory (153 MB average per pack-objects, up to 267 MB), next to 2,334 MB for Jenkins and 761 MB for ATS, 7,270 MB in all. Free memory went from 3,485 MB to 72 MB in 42 s once the children started. The full breakdown is in apache/trafficserver-ci#474. There were 32 kernel out-of-memory kills from Sep 2 to Oct 13, and 4 more on Oct 14. Fetching only the branch being built is a 37 MB pack, about 75 MB and about 5 s.
- The 1-minute clone timeout wrapped all three retries. A single fetch takes about 15 s when the box is quiet, so the cap only failed under the load above. Because the retries were inside it, they never got a real chance.
Recent master branch builds:
| Build | Children passed |
|---|---|
| #1423 | 8 of 29 |
| #1424 | 6 of 29 |
Done
- Raise the branch clone timeout from 1 minute to 10 (apache/trafficserver-ci#473)
- Fetch only the branch being built in the branch pipelines (apache/trafficserver-ci#474)
- Stop ATS reusing connections to the mirror (
server_session_sharing.match=noneon the/mirror/remap, controller config) - Cap the mirror at 16 httpd workers and 2 GB of memory (controller config)
To do
- Give each branch checkout retry its own timeout, with a random backoff between tries:
retry(3) { sleep; timeout(5) { checkout } }(apache/trafficserver-ci#476) - Add the same retry, timeout and backoff to the GitHub PR pipelines, which have none today (also in apache/trafficserver-ci#476)
- Add a timeout and retry to the trafficserver-ci checkout in the branch pipelines
- Apply the connection fix to the docs and Jenkins remaps as well, and save it in the config so it survives a restart
- Turn off ATS debug logging on the controller (about 3.3 GB an hour of
traffic.out) - Shrink the ATS RAM cache on the controller (1 GB today)
- Check the next branch build's fan-out against the new 2 GB mirror limit. 16 workers fetching all branches at the 153 MB average seen under load is about 16 x 153 = 2,450 MB, over the limit, so git will likely be killed inside the container until fetches are branch-only.
- Find out why freebsd, osx and osx-m1 fail in Clone at about 180 s ("3775 bytes of body are still expected")
- Fix the mirror webhook leaving thousands of finished
gitprocesses behind - Point the 10.1.x and 10.2.x branch pollers at their own branches (both watch 10.0.x)
- Free disk on the controller (90% full)
- File an ATS issue for reusing a backend connection the backend already closed
Lower priority
- Repack the mirror into one pack (about 20% faster fetches)
- Remove the duplicate
connect_attempts_timeoutand rename the deprecatedconnect_attempts_max_retries_down_serverin the controller's records.yaml
The pipeline PRs are in apache/trafficserver-ci, which has issues turned off, so this lives here. PRs for these items should mention this issue so they link back.
- 主要语言
- C++
- 星标
- 2k
- 派生
- 882
- 平均合并
- 3 天 21 小时
- 30 天内合并 PR
- 101
环境准备
- 没有 Dockerfile 或 Docker Compose 文件
- 没有 Pull Request 模板
- 阅读贡献指南
从这里开始
- 先读完整个 Issue，再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库，在一个分支上完成修改。
- 提交 Pull Request，并在描述里引用这个 Issue 编号。
apache/trafficserver 的其他 Issue
-
Network Performance
难度 5/5 一周以上 新手友好度 42/100
apache/trafficserver#13890 ·
维护者通常 2 天内回复
-
Uranium follow-ups from #13545可能已有人在做 @bneradt 于 4 天前认领。 未关闭AuTest Tests
apache/trafficserver#13877 · 已指派 1 人 ·
维护者通常 2 天内回复
-
traffic_ctl: apply --error-level to config reload results可能已有人在做 @brbzull0 于 12 天前认领。 未关闭traffic_ctl
apache/trafficserver#13832 · 已指派 1 人 ·
维护者通常 2 天内回复
-
header_rewrite: rm-destination after set-destination URL crashes traffic_server可能已有人在做 @moonchen 于 19 天前认领。 未关闭Bug Crash header_rewrite Plugins
apache/trafficserver#13800 · 已指派 1 人 ·
维护者通常 2 天内回复
-
ssl_multicert: entries without dest_ip can override the default certificate可能已有人在做 @toxicteddy00077 于 6 天前认领。 未关闭Configuration SSL
难度 4/5 3-5 天 新手友好度 48/100
apache/trafficserver#13798 · 2 条评论 · 已指派 1 人 ·
维护者通常 2 天内回复
查看 apache/trafficserver 的全部 Issue
相似的 Issue
-
难度 2/5 1-3 小时 新手友好度 76/100
sistemicorp/scripts#62 ·
-
难度 1/5 1 小时以内 新手友好度 72/100
维护者通常 1 天内回复
-
extended_stabilizer: `std::pow(2, p/2.)` in `scalar_t::to_complex()` takes ~40% of runtime on Linux可能已有人在做 @mehmetniyazikayi 今天认领。 未关闭
难度 2/5 1-3 小时 新手友好度 74/100
Qiskit/qiskit-aer#2502 ·
维护者通常 1 天内回复
-
Test failure on FreeBSD: #26 unexpected system_clock source可能已有人在做 关联的 PR 仍在进行中或已合并。 未关闭low priority rt_lib tests
难度 2/5 1-3 小时 新手友好度 67/100
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 65/100
RobLoach/raylib-cpp#440 · 2 条评论 ·