gVisor restore runs inside the actor's CPU limit, so wake time scales with 1/limit

Cerrado
  • ¿Algo no está bien?
#3,431 4 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
35/100
Tipo de issue
Nueva funcionalidad
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
go, linux
Área
backend, infrastructure

Línea de trabajo

Start by tracing the restore path and the existing checkpoint cgroup handling in ckptcgroup.go; the issue also points to internal/ocispec, sizing, and cgroup as relevant areas. Check how runsc restore is launched, when ingress opens, and how the sentry moves between cgroups. Done means restore completes outside the actor CPU cap and the sentry enters its sandbox cgroup before ingress opens, with the CPU ceiling policy decided.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

area/gvisor area/node kind/bug

We run v0.6.0 (gVisor nightly 2026-10-17) on dense AMD EPYC 9845 nodes (Zen 5c, 320 threads, Rocky 10, kernel 6.12). Small actors wake slowly, and the time is almost all worker_restore: download, unpack and mount are all 0 in atelet's restore timing breakdown. As far as I can tell, runsc restore runs in the actor's <uid>-_sandbox cgroup with cpu.max already set to the declared limit (the 100ms period from internal/ocispec, --cpu-num-from-quota). So the restore gets only the actor's share of a core.

A restore costs us about 0.15 CPU-seconds, and wake time follows that divided by the limit. Here's one counter actor at a time on an otherwise idle worker, 10 wakes per limit:

CPU limit wake p50 / p90 (s)
50m 9.0 / 9.6 (9 of 10 hit the router timeout)
100m 2.0 / 2.3
250m 0.62 / 0.68
500m 0.26 / 0.29
1000m 0.15 / 0.18
2000m 0.13 / 0.13

On a Xeon E-2288G the same 100m actor wakes in about 0.6s, so the cost is per-core speed multiplied by the cap.

Things we tried:

  • kernel.sched_cfs_bandwidth_slice_us=1000 takes 50m from 9s with failures to 3.8s with none, but does little above 100m.
  • A 10ms CFS period (patched ocispec, sizing and cgroup) was the same at 100m and 2-3x slower at 250m and 500m.
  • THP madvise made no difference.
  • Checkpoint compression is already none by default in this gVisor.

What I'd like to propose: restore outside the cap, then apply it. Checkpoint already does something like this. ckptcgroup.go moves the sentry into a sibling <uid>-_ckpt cgroup while it saves, copying cpu.max across. A restore could run in a sibling cgroup with no cpu.max (or a WorkerPool-level ceiling), then move the sentry into <uid>-_sandbox once restore completes, before ingress opens. With that, every limit would wake at roughly the 2000m figure.

The trade-off is that a restoring actor briefly uses more CPU than it declared, at most what the worker pod itself is allowed. If that's a concern, it could be opt-in per WorkerPool, or capped at something like max(limit, 1 core).

Is this something you'd take? If so, would you rather the ceiling be a WorkerPool field or a fixed rule?

Lenguaje dominante
Go
Estrellas
6.1k
Forks
676
Merge medio
2 d 2 h
PR fusionados (30 d)
522

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de agent-substrate/substrate

Todos los issues de agent-substrate/substrate

Issues similares

Más issues de Go

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.