Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

Branch build clones: mirror stalls, clone timeout and controller memory

Aperta
  • Qualcosa non va?
#13,839 0 commenti 0 reazioni 1 assegnatario Vedi su GitHub

I maintainer di solito rispondono entro 2 giorni

@bryancall ci sta già lavorando.

Dal 14/10/2026.

Valutazione

Questa issue non è ancora stata valutata.

Descrizione

CI

Branch builds keep failing in the Clone stage, and the controller keeps running out of memory. I dug into it on Oct 13 and 14, and there are three separate problems:

  1. ATS reused connections to the mirror after the mirror had closed them. Each reuse sat for 180 s before ATS retried. On Oct 13 there were 30 of these stalls on the mirror, 23 on the docs site and 8 on Jenkins.
  2. Branch builds fetch every branch at depth 1000. One fetch takes about 15 s and about 125 MB of private memory on the controller. A branch build starts about 30 children at once on a 4 CPU, 8 GB box with no swap, so the fetches overlap and it runs out of memory. At the first Jenkins kill on Oct 14 (12:43 PDT), the kernel's out-of-memory report had 21 fetches in flight using 3,317 MB of git memory (153 MB average per pack-objects, up to 267 MB), next to 2,334 MB for Jenkins and 761 MB for ATS, 7,270 MB in all. Free memory went from 3,485 MB to 72 MB in 42 s once the children started. The full breakdown is in apache/trafficserver-ci#474. There were 32 kernel out-of-memory kills from Sep 2 to Oct 13, and 4 more on Oct 14. Fetching only the branch being built is a 37 MB pack, about 75 MB and about 5 s.
  3. The 1-minute clone timeout wrapped all three retries. A single fetch takes about 15 s when the box is quiet, so the cap only failed under the load above. Because the retries were inside it, they never got a real chance.

Recent master branch builds:

Build Children passed
#1423 8 of 29
#1424 6 of 29
Done
  • Raise the branch clone timeout from 1 minute to 10 (apache/trafficserver-ci#473)
  • Fetch only the branch being built in the branch pipelines (apache/trafficserver-ci#474)
  • Stop ATS reusing connections to the mirror (server_session_sharing.match=none on the /mirror/ remap, controller config)
  • Cap the mirror at 16 httpd workers and 2 GB of memory (controller config)
To do
  • Give each branch checkout retry its own timeout, with a random backoff between tries: retry(3) { sleep; timeout(5) { checkout } } (apache/trafficserver-ci#476)
  • Add the same retry, timeout and backoff to the GitHub PR pipelines, which have none today (apache/trafficserver-ci#475)
  • Add a timeout and retry to the trafficserver-ci checkout in the branch pipelines
  • Apply the connection fix to the docs and Jenkins remaps as well, and save it in the config so it survives a restart
  • Turn off ATS debug logging on the controller (about 3.3 GB an hour of traffic.out)
  • Shrink the ATS RAM cache on the controller (1 GB today)
  • Check the next branch build's fan-out against the new 2 GB mirror limit. 16 workers fetching all branches at the 153 MB average seen under load is about 16 x 153 = 2,450 MB, over the limit, so git will likely be killed inside the container until fetches are branch-only.
  • Find out why freebsd, osx and osx-m1 fail in Clone at about 180 s ("3775 bytes of body are still expected")
  • Fix the mirror webhook leaving thousands of finished git processes behind
  • Point the 10.1.x and 10.2.x branch pollers at their own branches (both watch 10.0.x)
  • Free disk on the controller (90% full)
  • File an ATS issue for reusing a backend connection the backend already closed
Lower priority
  • Repack the mirror into one pack (about 20% faster fetches)
  • Remove the duplicate connect_attempts_timeout and rename the deprecated connect_attempts_max_retries_down_server in the controller's records.yaml

The pipeline PRs are in apache/trafficserver-ci, which has issues turned off, so this lives here. PRs for these items should mention this issue so they link back.

Lingua principale
C++
Stelle
2k
Fork
878
Merge medio
3g 15h
PR unite (30g)
96

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di apache/trafficserver

Tutte le issue di apache/trafficserver

Issue simili

Altre issue su C++

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.