CI perf (settings): merge queue builds 5 entries at a time — a 13-PR burst took 100 min to drain on a 21-min CI run (~21 min per PR past the 5th)

オープン
  • 何か問題がありますか？
#1,248 コメント 13 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

まだ誰も着手していません。

評価

難易度
2/5
見積もり時間
1時間未満
初心者へのやさしさ
12/100
issue の種類
リファクタリング
明瞭さ
おおむね明確
活発さ
活発
技術スタック
github-actions
領域
ci-cd

調査の方向性

The change is a GitHub merge-queue ruleset setting (ruleset 24668460, merge_queue rule, max_entries_to_build), not a file in the repository, so there is no code to open. Only a maintainer with admin access can decide and apply it, and the issue itself says the implementer routine cannot. Done means a human has made the decision and either applied the new value or closed the issue with a reason.

索引モデルが issue の本文から書いたものです。

説明

agent:needs-human agent:triaged ci-perf priority:p3

Filed by the CI profiler routine (observer only). This is a ruleset change, so a human has to decide; the implementer routine can't apply it.

Measurement

Window: 2026-10-08 08:17 → 2026-10-09 08:17 UTC. Sources: ruleset 24668460, 157 CI merge_group runs, job detail for 52 of them, and REST timelines of the 14 most recently merged PRs.

Current merge_queue rule: max_entries_to_build: 5, max_entries_to_merge: 5, min_entries_to_merge: 1, grouping_strategy: ALLGREEN, check_response_timeout_minutes: 120.

metric value
PRs merged / 24h 69
merge_group CI success, created → done, p50 / p90 23.4 / 46.1 min (n=80); 21.5 / 23.0 min since 04:17 (n=16)
Last enqueue → merge, p50 / p90 (14 PRs merged 03:14–08:01) 64 / 69 min
First enqueue → merge, p50 / p90 65 / 82 min

What happened in the 06:19–08:01 burst. 13 PRs were enqueued between 06:19 and 06:52: #1215, #1209, #1208, #1217, #1193, #1218, #1036, #1041, #1226, #1230, #1229, #1228, plus #1007. They were tested in waves of at most 5 CI runs, and each wave started only after the previous one finished:

wave created runs outcome
06:19–06:20 5 all cancelled. test (windows-latest, 1) failed in 37892957346 and evicted #1215; the other 4 were rebuilds (37892958452 etc.)
06:34–06:35 5 2 ok, 3 cancelled (coverage-docker (sbt) failure in 37894279963)
06:56 5 all ok → merged 06:56 / 07:19
07:17–07:19 5 all ok → merged 07:38–07:41
07:38–07:41 4 (+1 cancelled) merged 08:01

So the queue's throughput is 5 PRs per ~21 min (≈14/h). The 6th–13th PRs of a burst wait one or two full waves before their own run even starts. Of the 64-min p50, about 21 min is the PR's own run and ~40 min is waiting behind earlier waves.

Where the time goes

  • Per wave: test (windows-latest, 2) (Run tests 15.0 min p50 + Build 2.2), coverage-docker (sbt) → coverage-merge (#1225), and the Gradle e2e_redirect_gradle_build … gradle_hosted_b legs (13.4 min p50, #1171) all finish at 20–22 min. Queue wait for those jobs is ≤0.1 min (Linux/Windows) and 0.2 min p50 (macOS), so the waves are not runner-starved. The limit is the build depth.
  • Rebuild cost when an entry is evicted (52-run sample):
    • 15 runs cancelled purely as rebuilds behind an evicted entry: 4,311 job-min, 287 avg
    • 10 runs with a genuine failure: 4,110 job-min
    • 27 successful runs: 12,795 job-min, 474 avg
    • A deeper build window raises the first number.

Root cause

max_entries_to_build: 5 caps the speculative pipeline at 5 entries. With ALLGREEN groups of up to 5 and every wave taking about the same ~21 min, the queue advances in lock-step waves. During the agent bursts that now produce most merges, every PR past the 5th waits at least one extra full wave.

Proposed fix (ruleset 24668460, merge_queue rule)

  1. Raise max_entries_to_build 5 → 8, and keep max_entries_to_merge at 5 (or raise it to 8 too, so a fully green window lands in one merge).
  2. Leave grouping_strategy, timeouts and the required checks (ci-ok, clippy) unchanged.
  3. Roll back if any of these hold over the following 24h:
    • macOS job queue-wait p90 goes above ~5 min (it is 1.8 min now)
    • the merge_group eviction rate stays above ~20%. Then the extra rebuilds cost more than the latency buys; prefer landing the ci-janitor flake fixes (#1237, #1238, #1222, #1244) first.

Expected saving

  • Latency: ~21 min off enqueue → merge for every PR that is 6th–8th in line. In the window above that was 8 of 13 PRs. Over the day, assuming about a third of 69 merges arrive in bursts, that is ~20 PRs × ~21 min ≈ 7 PR-hours/day. Merge-queue p50 should fall from ~64 toward ~45 min during bursts.
  • Critical path per affected PR: −21 min (one wave).
  • Cost:
    • Each eviction during a burst can rebuild up to 7 entries instead of 4: ≈ +3 × 287 ≈ +860 job-min per eviction.
    • At ~5–8 burst evictions/day that is +4–7k job-min/day, ~11% of it macOS (51 mac-min per full run).
    • Peak concurrent macOS jobs from the queue would go from 50 (measured at 06:42 with 5 entries; 28.9 mac jobs per run) to roughly 80.
    • #1176 / #1234 (fewer e2e-macos legs) would offset most of that.

Coverage and risk

  • No test moves; every job still runs on every merge_group run.
  • Required check names are unchanged.
  • Main risk: macOS pool pressure. It is shared with push-to-main runs (42/day × ~56 mac-min) and every other merge_group run. Watch macOS queue wait on the CI performance dashboard after the change.
  • Second risk: more cancelled rebuild minutes while the eviction rate is still ~20–30%.

Effort

S (one ruleset field, reversible in seconds).

ROI

About 7: (≈21 critical-path min − ≈7 weighted for +5k job-min) × confidence 0.5 ÷ effort 1. Same scale as the dashboard.

主要言語
Rust
スター
9
フォーク
2
平均マージ
20時間 47分
マージ済み PR（30日）
681

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

SocketDev/socket-patch のほかの issue

SocketDev/socket-patch の issue をすべて見る

似ている issue

Rust の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。