CI perf (settings): slow Depot 16-core hosts since 10-13 15:20 add ~1.5–2.5 min to the merge-queue critical path (~0.6k Linux job-min/day)

Ouverte
  • Quelque chose ne va pas ?
#1,512 17 commentaires 0 réactions 0 personnes assignées Voir sur GitHub

Les mainteneurs répondent en général sous 1 jour

Personne n'a encore pris cette issue.

Évaluation

Difficulté
3/5
Temps estimé
Une demi-journée
Accessibilité débutants
15/100
Type d'issue
Bug
Clarté
Plutôt claire
Activité
Active
Stack technique
github-actions, rust
Domaine
devops

Piste de recherche

The likely repo-side change is the runs-on label for the coverage and clippy jobs in .github/workflows/ci.yml, currently depot-ubuntu-24.04-16. The first step is a human check of the Depot host health and the linked job runs, because the cause is in the runner fleet, not the code. Done means one measured CI batch on the new runner class shows a healthy start gap, with job names and the ci-ok check unchanged.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Description

agent:needs-human agent:triaged ci-perf priority:p3

Filed by the CI profiler routine. This needs a human: the cause is in the Depot runner fleet, not in the repo, so the implementer routine shouldn't pick it up as a code change.

Measurement

Window: 2026-10-13 13:11 → 2026-10-14 00:00 UTC. Sample: jobs from all 52 CI merge_group runs, 41 CI pull_request runs and 15 push runs (2,253 jobs, filter=all).

Two speeds of host on the same runner class. A job is counted as being on a "slow" host when the gap from job start to its first step is over 15 s. Healthy hosts start in about 6 s.

period (UTC) depot-ubuntu-24.04-16: share of slow-start jobs depot-ubuntu-24.04-4: share of slow-start jobs
13:00–15:20 21% (n=126, p50 gap 6 s) 23% (n=999, p50 6 s)
15:20–22:00 59% (n=90, p50 23.5 s) 70% (n=637, p50 23 s)
22:00–00:00 57% (n=42, p50 19.5 s) 24% (n=286, p50 6 s): recovered

The 4-core class recovered at about 22:00. The 16-core class, which runs clippy and coverage (both on the merge-queue critical path), did not.

coverage → Run tests with coverage step time (successful jobs):

fast host slow host
13:00–15:20 p50 332 s (n=29) –
15:20–22:00 p50 324 s (n=13) p50 424 s (n=17)
22:00+ 320 s (n=1) p50 406 s (n=7 of 8)

The inputs are the same, so the difference is the host. In the 23:49 batch the CI inputs differ by one PR each, and the host coverage TOTAL lines match within 0.02% (350,679 vs 350,622):

  • job 115602304980 on depot-s9bzhj8q5x: start gap 5 s, test step 320 s, job 5.7 min.
  • job 115602276746 on depot-xb3zx5n43w: start gap 22 s, test step 466 s, job 8.7 min.

clippy (also 16-core, and coverage waits on it) takes 1.3–1.4 min on fast hosts and 2.1–2.2 min on slow ones, for example 38486575290 compared with 38480343655.

Effect on landing PRs:

  • merge_group created → ci-ok: 7.4–9.1 min before 15:20 (n=14 non-evicted runs at 13:11–13:35). Since then it has been p50 9.5, max 12.2 (n=8 since 20:16). coverage was the last job before ci-ok in 7 of 8 of the runs since 22:49.
  • Enqueue → merge: p50 11.3 / p90 12.0 min (n=15). The 10-12 profiles had 8.0–8.8 at the same settings.
  • PR created → ci-ok: p50 9.9 / p90 12.1 (n=19 full runs). It was 7.4–7.6 on 10-12.

Where the time goes

Each slow-host coverage job loses about 15–20 s of start-up and 7–9 s of Checkout/Cache cargo/Install Rust (2–3 s each on fast hosts), plus ~80–145 s in the test step. Each slow-host clippy job loses about 50 s. When both land on slow hosts, which happened in 4 of the 8 runs since 22:49, the critical path grows by ~2–2.5 min.

Root cause

Part of the Depot 16-core pool is running ~25–45% slower on CPU and I/O than the rest. That started at about 15:20 UTC on 10-13. It isn't load from this repo: at 13:00 the repo ran ~700 jobs/hour with a 6 s start gap, and at 21:00 it ran ~33 jobs/hour with a 26 s gap. The 4-core pool recovered at 22:00 and the 16-core pool hasn't.

Proposed fix

  1. Human, S: check the Depot status page and the org's Depot dashboard for depot-ubuntu-24.04-16 host health since 2026-10-13 15:20 UTC. If nothing is posted, open a Depot support ticket with the two job links above, the runner names depot-xb3zx5n43w, depot-c59k7s930p, depot-f5768wcmwc, depot-jctrbg38bj, depot-cg4tbh0c9f, depot-g024vxvvxz and depot-n4hvg6kjz3, and the start-gap numbers.
  2. Only if Depot can't fix it within a day (repo side, S): in .github/workflows/ci.yml, move coverage and clippy from depot-ubuntu-24.04-16 to another class whose start gap is healthy (for example depot-ubuntu-24.04-32, or the 4-core class, which recovered). Measure one batch before committing to it. Don't rename either job: clippy is a required check, and coverage feeds ci-ok.

Expected saving

  • Merge-queue critical path: ~1.5 min p50, 9.5 → ~8. Up to 2.5 min when both jobs land on slow hosts.
  • PR feedback (created → ci-ok): ~1.5 min.
  • Job-minutes: ~2.5 min per coverage job and ~0.8 per clippy job on slow hosts, at ~57% slow placement. That is about 1.9 min per run, which is ~0.1k per day over 52 merge-queue runs and ~0.3k per day over ~150 PR runs. Adding the other 16-core jobs gives ~0.4–0.6k Linux job-min/day.

Coverage and risk

No test coverage changes. Option 1 doesn't touch the repo. Option 2 changes only runs-on labels, and keeps job names and ci-ok/clippy the same. Risk: another class may be slow too, which is why it should be measured first.

Effort

S. Option 1 is a ticket. Option 2 is a two-line runs-on change.

ROI

(0.5k job-min → 0.5, + 1.5 critical-path min + 1.5 PR-latency min) × confidence 0.5 ÷ effort 1 ≈ 1.75

Langage dominant
Rust
Étoiles
8
Forks
2
Merge moyen
18 h 47 min
PR mergées (30 j)
637

Préparer son environnement

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Autres issues de SocketDev/socket-patch

Toutes les issues de SocketDev/socket-patch

Issues similaires

Plus d'issues Rust

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.