CI perf (settings): merge queue builds 5 entries at a time — a 13-PR burst took 100 min to drain on a 21-min CI run (~21 min per PR past the 5th)

Offen
  • Stimmt etwas nicht?
#1,248 13 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen

Maintainer antworten meist innerhalb von 1 Tag

Dieses Issue hat noch niemand übernommen.

Bewertung

Schwierigkeit
2/5
Geschätzter Aufwand
Unter einer Stunde
Anfängerfreundlichkeit
12/100
Issue-Typ
Refactoring
Klarheit
Größtenteils klar
Aktivitätsstatus
Aktiv
Tech-Stack
github-actions
Bereich
ci-cd

Rechercherichtung

The change is a GitHub merge-queue ruleset setting (ruleset 24668460, merge_queue rule, max_entries_to_build), not a file in the repository, so there is no code to open. Only a maintainer with admin access can decide and apply it, and the issue itself says the implementer routine cannot. Done means a human has made the decision and either applied the new value or closed the issue with a reason.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Beschreibung

agent:needs-human agent:triaged ci-perf priority:p3

Filed by the CI profiler routine (observer only). This is a ruleset change, so a human has to decide; the implementer routine can't apply it.

Measurement

Window: 2026-10-08 08:17 → 2026-10-09 08:17 UTC. Sources: ruleset 24668460, 157 CI merge_group runs, job detail for 52 of them, and REST timelines of the 14 most recently merged PRs.

Current merge_queue rule: max_entries_to_build: 5, max_entries_to_merge: 5, min_entries_to_merge: 1, grouping_strategy: ALLGREEN, check_response_timeout_minutes: 120.

metric value
PRs merged / 24h 69
merge_group CI success, created → done, p50 / p90 23.4 / 46.1 min (n=80); 21.5 / 23.0 min since 04:17 (n=16)
Last enqueue → merge, p50 / p90 (14 PRs merged 03:14–08:01) 64 / 69 min
First enqueue → merge, p50 / p90 65 / 82 min

What happened in the 06:19–08:01 burst. 13 PRs were enqueued between 06:19 and 06:52: #1215, #1209, #1208, #1217, #1193, #1218, #1036, #1041, #1226, #1230, #1229, #1228, plus #1007. They were tested in waves of at most 5 CI runs, and each wave started only after the previous one finished:

wave created runs outcome
06:19–06:20 5 all cancelled. test (windows-latest, 1) failed in 37892957346 and evicted #1215; the other 4 were rebuilds (37892958452 etc.)
06:34–06:35 5 2 ok, 3 cancelled (coverage-docker (sbt) failure in 37894279963)
06:56 5 all ok → merged 06:56 / 07:19
07:17–07:19 5 all ok → merged 07:38–07:41
07:38–07:41 4 (+1 cancelled) merged 08:01

So the queue's throughput is 5 PRs per ~21 min (≈14/h). The 6th–13th PRs of a burst wait one or two full waves before their own run even starts. Of the 64-min p50, about 21 min is the PR's own run and ~40 min is waiting behind earlier waves.

Where the time goes

  • Per wave: test (windows-latest, 2) (Run tests 15.0 min p50 + Build 2.2), coverage-docker (sbt) → coverage-merge (#1225), and the Gradle e2e_redirect_gradle_build … gradle_hosted_b legs (13.4 min p50, #1171) all finish at 20–22 min. Queue wait for those jobs is ≤0.1 min (Linux/Windows) and 0.2 min p50 (macOS), so the waves are not runner-starved. The limit is the build depth.
  • Rebuild cost when an entry is evicted (52-run sample):
    • 15 runs cancelled purely as rebuilds behind an evicted entry: 4,311 job-min, 287 avg
    • 10 runs with a genuine failure: 4,110 job-min
    • 27 successful runs: 12,795 job-min, 474 avg
    • A deeper build window raises the first number.

Root cause

max_entries_to_build: 5 caps the speculative pipeline at 5 entries. With ALLGREEN groups of up to 5 and every wave taking about the same ~21 min, the queue advances in lock-step waves. During the agent bursts that now produce most merges, every PR past the 5th waits at least one extra full wave.

Proposed fix (ruleset 24668460, merge_queue rule)

  1. Raise max_entries_to_build 5 → 8, and keep max_entries_to_merge at 5 (or raise it to 8 too, so a fully green window lands in one merge).
  2. Leave grouping_strategy, timeouts and the required checks (ci-ok, clippy) unchanged.
  3. Roll back if any of these hold over the following 24h:
    • macOS job queue-wait p90 goes above ~5 min (it is 1.8 min now)
    • the merge_group eviction rate stays above ~20%. Then the extra rebuilds cost more than the latency buys; prefer landing the ci-janitor flake fixes (#1237, #1238, #1222, #1244) first.

Expected saving

  • Latency: ~21 min off enqueue → merge for every PR that is 6th–8th in line. In the window above that was 8 of 13 PRs. Over the day, assuming about a third of 69 merges arrive in bursts, that is ~20 PRs × ~21 min ≈ 7 PR-hours/day. Merge-queue p50 should fall from ~64 toward ~45 min during bursts.
  • Critical path per affected PR: −21 min (one wave).
  • Cost:
    • Each eviction during a burst can rebuild up to 7 entries instead of 4: ≈ +3 × 287 ≈ +860 job-min per eviction.
    • At ~5–8 burst evictions/day that is +4–7k job-min/day, ~11% of it macOS (51 mac-min per full run).
    • Peak concurrent macOS jobs from the queue would go from 50 (measured at 06:42 with 5 entries; 28.9 mac jobs per run) to roughly 80.
    • #1176 / #1234 (fewer e2e-macos legs) would offset most of that.

Coverage and risk

  • No test moves; every job still runs on every merge_group run.
  • Required check names are unchanged.
  • Main risk: macOS pool pressure. It is shared with push-to-main runs (42/day × ~56 mac-min) and every other merge_group run. Watch macOS queue wait on the CI performance dashboard after the change.
  • Second risk: more cancelled rebuild minutes while the eviction rate is still ~20–30%.

Effort

S (one ruleset field, reversible in seconds).

ROI

About 7: (≈21 critical-path min − ≈7 weighted for +5k job-min) × confidence 0.5 ÷ effort 1. Same scale as the dashboard.

Vorherrschende Sprache
Rust
Sterne
9
Forks
2
Ø Merge
18 Std. 47 Min.
Gemergte PRs (30 T.)
637

Entwicklungsumgebung

Erste Schritte

  1. Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
  2. Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
  3. Forken Sie das Repository und arbeiten Sie in einem Branch.
  4. Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.

Mehr aus SocketDev/socket-patch

Alle Issues in SocketDev/socket-patch

Ähnliche Issues

Weitere Issues zu Rust

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.