Hacktoberfest 2026: những issue maintainer đã đánh dấu cho tháng Mười, đang mở và phù hợp người mới. Xem issue Hacktoberfest

Branch build clones: mirror stalls, clone timeout and controller memory

Đang mở
  • Có gì không ổn?
#13,839 0 bình luận 0 reaction 1 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 2 ngày

@bryancall đang làm issue này rồi.

Từ ngày 14/10/2026.

Đánh giá

Issue này chưa được đánh giá.

Mô tả

CI

Branch builds keep failing in the Clone stage, and the controller keeps running out of memory. I dug into it on Oct 13 and 14, and there are three separate problems:

  1. ATS reused connections to the mirror after the mirror had closed them. Each reuse sat for 180 s before ATS retried. On Oct 13 there were 30 of these stalls on the mirror, 23 on the docs site and 8 on Jenkins.
  2. Branch builds fetch every branch at depth 1000. One fetch takes about 15 s and about 125 MB of private memory on the controller. A branch build starts about 30 children at once on a 4 CPU, 8 GB box with no swap, so the fetches overlap and it runs out of memory. At the first Jenkins kill on Oct 14 (12:43 PDT), the kernel's out-of-memory report had 21 fetches in flight using 3,317 MB of git memory (153 MB average per pack-objects, up to 267 MB), next to 2,334 MB for Jenkins and 761 MB for ATS, 7,270 MB in all. Free memory went from 3,485 MB to 72 MB in 42 s once the children started. The full breakdown is in apache/trafficserver-ci#474. There were 32 kernel out-of-memory kills from Sep 2 to Oct 13, and 4 more on Oct 14. Fetching only the branch being built is a 37 MB pack, about 75 MB and about 5 s.
  3. The 1-minute clone timeout wrapped all three retries. A single fetch takes about 15 s when the box is quiet, so the cap only failed under the load above. Because the retries were inside it, they never got a real chance.

Recent master branch builds:

Build Children passed
#1423 8 of 29
#1424 6 of 29
Done
  • Raise the branch clone timeout from 1 minute to 10 (apache/trafficserver-ci#473)
  • Fetch only the branch being built in the branch pipelines (apache/trafficserver-ci#474)
  • Stop ATS reusing connections to the mirror (server_session_sharing.match=none on the /mirror/ remap, controller config)
  • Cap the mirror at 16 httpd workers and 2 GB of memory (controller config)
To do
  • Give each branch checkout retry its own timeout, with a random backoff between tries: retry(3) { sleep; timeout(5) { checkout } } (apache/trafficserver-ci#476)
  • Add the same retry, timeout and backoff to the GitHub PR pipelines, which have none today (also in apache/trafficserver-ci#476)
  • Add a timeout and retry to the trafficserver-ci checkout in the branch pipelines
  • Apply the connection fix to the docs and Jenkins remaps as well, and save it in the config so it survives a restart
  • Turn off ATS debug logging on the controller (about 3.3 GB an hour of traffic.out)
  • Shrink the ATS RAM cache on the controller (1 GB today)
  • Check the next branch build's fan-out against the new 2 GB mirror limit. 16 workers fetching all branches at the 153 MB average seen under load is about 16 x 153 = 2,450 MB, over the limit, so git will likely be killed inside the container until fetches are branch-only.
  • Find out why freebsd, osx and osx-m1 fail in Clone at about 180 s ("3775 bytes of body are still expected")
  • Fix the mirror webhook leaving thousands of finished git processes behind
  • Point the 10.1.x and 10.2.x branch pollers at their own branches (both watch 10.0.x)
  • Free disk on the controller (90% full)
  • File an ATS issue for reusing a backend connection the backend already closed
Lower priority
  • Repack the mirror into one pack (about 20% faster fetches)
  • Remove the duplicate connect_attempts_timeout and rename the deprecated connect_attempts_max_retries_down_server in the controller's records.yaml

The pipeline PRs are in apache/trafficserver-ci, which has issues turned off, so this lives here. PRs for these items should mention this issue so they link back.

Ngôn ngữ chính
C++
Star
2k
Fork
878
Merge trung bình
3 ngày 9 giờ
Pull request đã merge (30 ngày)
91

Chuẩn bị môi trường

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của apache/trafficserver

Tất cả issue của apache/trafficserver

Issue tương tự

Thêm issue về C++

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.