gVisor restore runs inside the actor's CPU limit, so wake time scales with 1/limit

クローズ
  • 何か問題がありますか？
#3,431 コメント 4 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

まだ誰も着手していません。

評価

難易度
5/5
見積もり時間
1週間以上
初心者へのやさしさ
35/100
issue の種類
機能追加
明瞭さ
おおむね明確
活発さ
活発
技術スタック
go, linux
領域
backend, infrastructure

調査の方向性

Start by tracing the restore path and the existing checkpoint cgroup handling in ckptcgroup.go; the issue also points to internal/ocispec, sizing, and cgroup as relevant areas. Check how runsc restore is launched, when ingress opens, and how the sentry moves between cgroups. Done means restore completes outside the actor CPU cap and the sentry enters its sandbox cgroup before ingress opens, with the CPU ceiling policy decided.

索引モデルが issue の本文から書いたものです。

説明

area/gvisor area/node kind/bug

We run v0.6.0 (gVisor nightly 2026-10-17) on dense AMD EPYC 9845 nodes (Zen 5c, 320 threads, Rocky 10, kernel 6.12). Small actors wake slowly, and the time is almost all worker_restore: download, unpack and mount are all 0 in atelet's restore timing breakdown. As far as I can tell, runsc restore runs in the actor's <uid>-_sandbox cgroup with cpu.max already set to the declared limit (the 100ms period from internal/ocispec, --cpu-num-from-quota). So the restore gets only the actor's share of a core.

A restore costs us about 0.15 CPU-seconds, and wake time follows that divided by the limit. Here's one counter actor at a time on an otherwise idle worker, 10 wakes per limit:

CPU limit wake p50 / p90 (s)
50m 9.0 / 9.6 (9 of 10 hit the router timeout)
100m 2.0 / 2.3
250m 0.62 / 0.68
500m 0.26 / 0.29
1000m 0.15 / 0.18
2000m 0.13 / 0.13

On a Xeon E-2288G the same 100m actor wakes in about 0.6s, so the cost is per-core speed multiplied by the cap.

Things we tried:

  • kernel.sched_cfs_bandwidth_slice_us=1000 takes 50m from 9s with failures to 3.8s with none, but does little above 100m.
  • A 10ms CFS period (patched ocispec, sizing and cgroup) was the same at 100m and 2-3x slower at 250m and 500m.
  • THP madvise made no difference.
  • Checkpoint compression is already none by default in this gVisor.

What I'd like to propose: restore outside the cap, then apply it. Checkpoint already does something like this. ckptcgroup.go moves the sentry into a sibling <uid>-_ckpt cgroup while it saves, copying cpu.max across. A restore could run in a sibling cgroup with no cpu.max (or a WorkerPool-level ceiling), then move the sentry into <uid>-_sandbox once restore completes, before ingress opens. With that, every limit would wake at roughly the 2000m figure.

The trade-off is that a restoring actor briefly uses more CPU than it declared, at most what the worker pod itself is allowed. If that's a concern, it could be opt-in per WorkerPool, or capped at something like max(limit, 1 core).

Is this something you'd take? If so, would you rather the ceiling be a WorkerPool field or a fixed rule?

主要言語
Go
スター
6.1k
フォーク
676
平均マージ
2日 2時間
マージ済み PR（30日）
522

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

agent-substrate/substrate のほかの issue

agent-substrate/substrate の issue をすべて見る

似ている issue

Go の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。