CI perf (settings): slow Depot 16-core hosts since 10-13 15:20 add ~1.5–2.5 min to the merge-queue critical path (~0.6k Linux job-min/day)

Offen
  • Stimmt etwas nicht?
#1,512 17 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen

Maintainer antworten meist innerhalb von 1 Tag

Dieses Issue hat noch niemand übernommen.

Bewertung

Schwierigkeit
3/5
Geschätzter Aufwand
Ein halber Tag
Anfängerfreundlichkeit
15/100
Issue-Typ
Bug
Klarheit
Größtenteils klar
Aktivitätsstatus
Aktiv
Tech-Stack
github-actions, rust
Bereich
devops

Rechercherichtung

The likely repo-side change is the runs-on label for the coverage and clippy jobs in .github/workflows/ci.yml, currently depot-ubuntu-24.04-16. The first step is a human check of the Depot host health and the linked job runs, because the cause is in the runner fleet, not the code. Done means one measured CI batch on the new runner class shows a healthy start gap, with job names and the ci-ok check unchanged.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Beschreibung

agent:needs-human agent:triaged ci-perf priority:p3

Filed by the CI profiler routine. This needs a human: the cause is in the Depot runner fleet, not in the repo, so the implementer routine shouldn't pick it up as a code change.

Measurement

Window: 2026-10-13 13:11 → 2026-10-14 00:00 UTC. Sample: jobs from all 52 CI merge_group runs, 41 CI pull_request runs and 15 push runs (2,253 jobs, filter=all).

Two speeds of host on the same runner class. A job is counted as being on a "slow" host when the gap from job start to its first step is over 15 s. Healthy hosts start in about 6 s.

period (UTC) depot-ubuntu-24.04-16: share of slow-start jobs depot-ubuntu-24.04-4: share of slow-start jobs
13:00–15:20 21% (n=126, p50 gap 6 s) 23% (n=999, p50 6 s)
15:20–22:00 59% (n=90, p50 23.5 s) 70% (n=637, p50 23 s)
22:00–00:00 57% (n=42, p50 19.5 s) 24% (n=286, p50 6 s): recovered

The 4-core class recovered at about 22:00. The 16-core class, which runs clippy and coverage (both on the merge-queue critical path), did not.

coverage → Run tests with coverage step time (successful jobs):

fast host slow host
13:00–15:20 p50 332 s (n=29) –
15:20–22:00 p50 324 s (n=13) p50 424 s (n=17)
22:00+ 320 s (n=1) p50 406 s (n=7 of 8)

The inputs are the same, so the difference is the host. In the 23:49 batch the CI inputs differ by one PR each, and the host coverage TOTAL lines match within 0.02% (350,679 vs 350,622):

  • job 115602304980 on depot-s9bzhj8q5x: start gap 5 s, test step 320 s, job 5.7 min.
  • job 115602276746 on depot-xb3zx5n43w: start gap 22 s, test step 466 s, job 8.7 min.

clippy (also 16-core, and coverage waits on it) takes 1.3–1.4 min on fast hosts and 2.1–2.2 min on slow ones, for example 38486575290 compared with 38480343655.

Effect on landing PRs:

  • merge_group created → ci-ok: 7.4–9.1 min before 15:20 (n=14 non-evicted runs at 13:11–13:35). Since then it has been p50 9.5, max 12.2 (n=8 since 20:16). coverage was the last job before ci-ok in 7 of 8 of the runs since 22:49.
  • Enqueue → merge: p50 11.3 / p90 12.0 min (n=15). The 10-12 profiles had 8.0–8.8 at the same settings.
  • PR created → ci-ok: p50 9.9 / p90 12.1 (n=19 full runs). It was 7.4–7.6 on 10-12.

Where the time goes

Each slow-host coverage job loses about 15–20 s of start-up and 7–9 s of Checkout/Cache cargo/Install Rust (2–3 s each on fast hosts), plus ~80–145 s in the test step. Each slow-host clippy job loses about 50 s. When both land on slow hosts, which happened in 4 of the 8 runs since 22:49, the critical path grows by ~2–2.5 min.

Root cause

Part of the Depot 16-core pool is running ~25–45% slower on CPU and I/O than the rest. That started at about 15:20 UTC on 10-13. It isn't load from this repo: at 13:00 the repo ran ~700 jobs/hour with a 6 s start gap, and at 21:00 it ran ~33 jobs/hour with a 26 s gap. The 4-core pool recovered at 22:00 and the 16-core pool hasn't.

Proposed fix

  1. Human, S: check the Depot status page and the org's Depot dashboard for depot-ubuntu-24.04-16 host health since 2026-10-13 15:20 UTC. If nothing is posted, open a Depot support ticket with the two job links above, the runner names depot-xb3zx5n43w, depot-c59k7s930p, depot-f5768wcmwc, depot-jctrbg38bj, depot-cg4tbh0c9f, depot-g024vxvvxz and depot-n4hvg6kjz3, and the start-gap numbers.
  2. Only if Depot can't fix it within a day (repo side, S): in .github/workflows/ci.yml, move coverage and clippy from depot-ubuntu-24.04-16 to another class whose start gap is healthy (for example depot-ubuntu-24.04-32, or the 4-core class, which recovered). Measure one batch before committing to it. Don't rename either job: clippy is a required check, and coverage feeds ci-ok.

Expected saving

  • Merge-queue critical path: ~1.5 min p50, 9.5 → ~8. Up to 2.5 min when both jobs land on slow hosts.
  • PR feedback (created → ci-ok): ~1.5 min.
  • Job-minutes: ~2.5 min per coverage job and ~0.8 per clippy job on slow hosts, at ~57% slow placement. That is about 1.9 min per run, which is ~0.1k per day over 52 merge-queue runs and ~0.3k per day over ~150 PR runs. Adding the other 16-core jobs gives ~0.4–0.6k Linux job-min/day.

Coverage and risk

No test coverage changes. Option 1 doesn't touch the repo. Option 2 changes only runs-on labels, and keeps job names and ci-ok/clippy the same. Risk: another class may be slow too, which is why it should be measured first.

Effort

S. Option 1 is a ticket. Option 2 is a two-line runs-on change.

ROI

(0.5k job-min → 0.5, + 1.5 critical-path min + 1.5 PR-latency min) × confidence 0.5 ÷ effort 1 ≈ 1.75

Vorherrschende Sprache
Rust
Sterne
8
Forks
2
Ø Merge
18 Std. 47 Min.
Gemergte PRs (30 T.)
637

Entwicklungsumgebung

Erste Schritte

  1. Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
  2. Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
  3. Forken Sie das Repository und arbeiten Sie in einem Branch.
  4. Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.

Mehr aus SocketDev/socket-patch

Alle Issues in SocketDev/socket-patch

Ähnliche Issues

Weitere Issues zu Rust

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.