gVisor restore runs inside the actor's CPU limit, so wake time scales with 1/limit

Chiusa
  • Qualcosa non va?
#3,431 4 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
35/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
go, linux
Ambito
backend, infrastructure

Direzione di ricerca

Start by tracing the restore path and the existing checkpoint cgroup handling in ckptcgroup.go; the issue also points to internal/ocispec, sizing, and cgroup as relevant areas. Check how runsc restore is launched, when ingress opens, and how the sentry moves between cgroups. Done means restore completes outside the actor CPU cap and the sentry enters its sandbox cgroup before ingress opens, with the CPU ceiling policy decided.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

area/gvisor area/node kind/bug

We run v0.6.0 (gVisor nightly 2026-10-17) on dense AMD EPYC 9845 nodes (Zen 5c, 320 threads, Rocky 10, kernel 6.12). Small actors wake slowly, and the time is almost all worker_restore: download, unpack and mount are all 0 in atelet's restore timing breakdown. As far as I can tell, runsc restore runs in the actor's <uid>-_sandbox cgroup with cpu.max already set to the declared limit (the 100ms period from internal/ocispec, --cpu-num-from-quota). So the restore gets only the actor's share of a core.

A restore costs us about 0.15 CPU-seconds, and wake time follows that divided by the limit. Here's one counter actor at a time on an otherwise idle worker, 10 wakes per limit:

CPU limit wake p50 / p90 (s)
50m 9.0 / 9.6 (9 of 10 hit the router timeout)
100m 2.0 / 2.3
250m 0.62 / 0.68
500m 0.26 / 0.29
1000m 0.15 / 0.18
2000m 0.13 / 0.13

On a Xeon E-2288G the same 100m actor wakes in about 0.6s, so the cost is per-core speed multiplied by the cap.

Things we tried:

  • kernel.sched_cfs_bandwidth_slice_us=1000 takes 50m from 9s with failures to 3.8s with none, but does little above 100m.
  • A 10ms CFS period (patched ocispec, sizing and cgroup) was the same at 100m and 2-3x slower at 250m and 500m.
  • THP madvise made no difference.
  • Checkpoint compression is already none by default in this gVisor.

What I'd like to propose: restore outside the cap, then apply it. Checkpoint already does something like this. ckptcgroup.go moves the sentry into a sibling <uid>-_ckpt cgroup while it saves, copying cpu.max across. A restore could run in a sibling cgroup with no cpu.max (or a WorkerPool-level ceiling), then move the sentry into <uid>-_sandbox once restore completes, before ingress opens. With that, every limit would wake at roughly the 2000m figure.

The trade-off is that a restoring actor briefly uses more CPU than it declared, at most what the worker pod itself is allowed. If that's a concern, it could be opt-in per WorkerPool, or capped at something like max(limit, 1 core).

Is this something you'd take? If so, would you rather the ceiling be a WorkerPool field or a fixed rule?

Lingua principale
Go
Stelle
5.9k
Fork
637
Merge medio
2g 1h
PR unite (30g)
523

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di agent-substrate/substrate

Tutte le issue di agent-substrate/substrate

Issue simili

Altre issue su Go

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.