gVisor restore runs inside the actor's CPU limit, so wake time scales with 1/limit

Open
  • Is something wrong?
#3,431 4 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
go, linux
Domain
backend, infrastructure

Research direction

Start by tracing the restore path and the existing checkpoint cgroup handling in ckptcgroup.go; the issue also points to internal/ocispec, sizing, and cgroup as relevant areas. Check how runsc restore is launched, when ingress opens, and how the sentry moves between cgroups. Done means restore completes outside the actor CPU cap and the sentry enters its sandbox cgroup before ingress opens, with the CPU ceiling policy decided.

Written by the indexing model from the issue text.

Description

area/gvisor area/node kind/bug

We run v0.6.0 (gVisor nightly 2026-10-17) on dense AMD EPYC 9845 nodes (Zen 5c, 320 threads, Rocky 10, kernel 6.12). Small actors wake slowly, and the time is almost all worker_restore: download, unpack and mount are all 0 in atelet's restore timing breakdown. As far as I can tell, runsc restore runs in the actor's <uid>-_sandbox cgroup with cpu.max already set to the declared limit (the 100ms period from internal/ocispec, --cpu-num-from-quota). So the restore gets only the actor's share of a core.

A restore costs us about 0.15 CPU-seconds, and wake time follows that divided by the limit. Here's one counter actor at a time on an otherwise idle worker, 10 wakes per limit:

CPU limit wake p50 / p90 (s)
50m 9.0 / 9.6 (9 of 10 hit the router timeout)
100m 2.0 / 2.3
250m 0.62 / 0.68
500m 0.26 / 0.29
1000m 0.15 / 0.18
2000m 0.13 / 0.13

On a Xeon E-2288G the same 100m actor wakes in about 0.6s, so the cost is per-core speed multiplied by the cap.

Things we tried:

  • kernel.sched_cfs_bandwidth_slice_us=1000 takes 50m from 9s with failures to 3.8s with none, but does little above 100m.
  • A 10ms CFS period (patched ocispec, sizing and cgroup) was the same at 100m and 2-3x slower at 250m and 500m.
  • THP madvise made no difference.
  • Checkpoint compression is already none by default in this gVisor.

What I'd like to propose: restore outside the cap, then apply it. Checkpoint already does something like this. ckptcgroup.go moves the sentry into a sibling <uid>-_ckpt cgroup while it saves, copying cpu.max across. A restore could run in a sibling cgroup with no cpu.max (or a WorkerPool-level ceiling), then move the sentry into <uid>-_sandbox once restore completes, before ingress opens. With that, every limit would wake at roughly the 2000m figure.

The trade-off is that a restoring actor briefly uses more CPU than it declared, at most what the worker pod itself is allowed. If that's a concern, it could be opt-in per WorkerPool, or capped at something like max(limit, 1 core).

Is this something you'd take? If so, would you rather the ceiling be a WorkerPool field or a fixed rule?

Dominant language
Go
Stars
5.9k
Forks
637
Avg merge
2d 1h
Merged PRs (30d)
523

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from agent-substrate/substrate

All issues in agent-substrate/substrate

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.