CI perf (settings): slow Depot 16-core hosts since 10-13 15:20 add ~1.5–2.5 min to the merge-queue critical path (~0.6k Linux job-min/day)

Abierto
  • ¿Algo no está bien?
#1,512 17 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
3/5
Tiempo estimado
Medio día
Aptitud para principiantes
15/100
Tipo de issue
Error
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
github-actions, rust
Área
devops

Línea de trabajo

The likely repo-side change is the runs-on label for the coverage and clippy jobs in .github/workflows/ci.yml, currently depot-ubuntu-24.04-16. The first step is a human check of the Depot host health and the linked job runs, because the cause is in the runner fleet, not the code. Done means one measured CI batch on the new runner class shows a healthy start gap, with job names and the ci-ok check unchanged.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

agent:needs-human agent:triaged ci-perf priority:p3

Filed by the CI profiler routine. This needs a human: the cause is in the Depot runner fleet, not in the repo, so the implementer routine shouldn't pick it up as a code change.

Measurement

Window: 2026-10-13 13:11 → 2026-10-14 00:00 UTC. Sample: jobs from all 52 CI merge_group runs, 41 CI pull_request runs and 15 push runs (2,253 jobs, filter=all).

Two speeds of host on the same runner class. A job is counted as being on a "slow" host when the gap from job start to its first step is over 15 s. Healthy hosts start in about 6 s.

period (UTC) depot-ubuntu-24.04-16: share of slow-start jobs depot-ubuntu-24.04-4: share of slow-start jobs
13:00–15:20 21% (n=126, p50 gap 6 s) 23% (n=999, p50 6 s)
15:20–22:00 59% (n=90, p50 23.5 s) 70% (n=637, p50 23 s)
22:00–00:00 57% (n=42, p50 19.5 s) 24% (n=286, p50 6 s): recovered

The 4-core class recovered at about 22:00. The 16-core class, which runs clippy and coverage (both on the merge-queue critical path), did not.

coverage → Run tests with coverage step time (successful jobs):

fast host slow host
13:00–15:20 p50 332 s (n=29) –
15:20–22:00 p50 324 s (n=13) p50 424 s (n=17)
22:00+ 320 s (n=1) p50 406 s (n=7 of 8)

The inputs are the same, so the difference is the host. In the 23:49 batch the CI inputs differ by one PR each, and the host coverage TOTAL lines match within 0.02% (350,679 vs 350,622):

  • job 115602304980 on depot-s9bzhj8q5x: start gap 5 s, test step 320 s, job 5.7 min.
  • job 115602276746 on depot-xb3zx5n43w: start gap 22 s, test step 466 s, job 8.7 min.

clippy (also 16-core, and coverage waits on it) takes 1.3–1.4 min on fast hosts and 2.1–2.2 min on slow ones, for example 38486575290 compared with 38480343655.

Effect on landing PRs:

  • merge_group created → ci-ok: 7.4–9.1 min before 15:20 (n=14 non-evicted runs at 13:11–13:35). Since then it has been p50 9.5, max 12.2 (n=8 since 20:16). coverage was the last job before ci-ok in 7 of 8 of the runs since 22:49.
  • Enqueue → merge: p50 11.3 / p90 12.0 min (n=15). The 10-12 profiles had 8.0–8.8 at the same settings.
  • PR created → ci-ok: p50 9.9 / p90 12.1 (n=19 full runs). It was 7.4–7.6 on 10-12.

Where the time goes

Each slow-host coverage job loses about 15–20 s of start-up and 7–9 s of Checkout/Cache cargo/Install Rust (2–3 s each on fast hosts), plus ~80–145 s in the test step. Each slow-host clippy job loses about 50 s. When both land on slow hosts, which happened in 4 of the 8 runs since 22:49, the critical path grows by ~2–2.5 min.

Root cause

Part of the Depot 16-core pool is running ~25–45% slower on CPU and I/O than the rest. That started at about 15:20 UTC on 10-13. It isn't load from this repo: at 13:00 the repo ran ~700 jobs/hour with a 6 s start gap, and at 21:00 it ran ~33 jobs/hour with a 26 s gap. The 4-core pool recovered at 22:00 and the 16-core pool hasn't.

Proposed fix

  1. Human, S: check the Depot status page and the org's Depot dashboard for depot-ubuntu-24.04-16 host health since 2026-10-13 15:20 UTC. If nothing is posted, open a Depot support ticket with the two job links above, the runner names depot-xb3zx5n43w, depot-c59k7s930p, depot-f5768wcmwc, depot-jctrbg38bj, depot-cg4tbh0c9f, depot-g024vxvvxz and depot-n4hvg6kjz3, and the start-gap numbers.
  2. Only if Depot can't fix it within a day (repo side, S): in .github/workflows/ci.yml, move coverage and clippy from depot-ubuntu-24.04-16 to another class whose start gap is healthy (for example depot-ubuntu-24.04-32, or the 4-core class, which recovered). Measure one batch before committing to it. Don't rename either job: clippy is a required check, and coverage feeds ci-ok.

Expected saving

  • Merge-queue critical path: ~1.5 min p50, 9.5 → ~8. Up to 2.5 min when both jobs land on slow hosts.
  • PR feedback (created → ci-ok): ~1.5 min.
  • Job-minutes: ~2.5 min per coverage job and ~0.8 per clippy job on slow hosts, at ~57% slow placement. That is about 1.9 min per run, which is ~0.1k per day over 52 merge-queue runs and ~0.3k per day over ~150 PR runs. Adding the other 16-core jobs gives ~0.4–0.6k Linux job-min/day.

Coverage and risk

No test coverage changes. Option 1 doesn't touch the repo. Option 2 changes only runs-on labels, and keeps job names and ci-ok/clippy the same. Risk: another class may be slow too, which is why it should be measured first.

Effort

S. Option 1 is a ticket. Option 2 is a two-line runs-on change.

ROI

(0.5k job-min → 0.5, + 1.5 critical-path min + 1.5 PR-latency min) × confidence 0.5 ÷ effort 1 ≈ 1.75

Lenguaje dominante
Rust
Estrellas
9
Forks
2
Merge medio
20 h 47 min
PR fusionados (30 d)
681

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de SocketDev/socket-patch

Todos los issues de SocketDev/socket-patch

Issues similares

Más issues de Rust

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.