[Bug]: A template without resource limits is admitted with no resource check, and a worker without limits registers the node's allocatable as its capacity

Open
  • Is something wrong?
#3,479 3 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
go, kubernetes
Domain
backend-api-design, infrastructure

Research direction

Start by tracing template create/derive validation, WorkerPool worker-limit handling, scheduler admission checks, and worker capacity registration; the issue does not name specific files or tests. Review the API guide alongside those entry points to understand the documented per-worker budget. Done means missing limits are handled consistently, registered capacity is logged, and the API guide explains the behavior.

Written by the indexing model from the issue text.

Description

area/api-machinery area/scheduling kind/bug
What happened?

Two defaults combine so that no admission control runs, with nothing reporting it.

Default Effect
A template with empty resources.limits is accepted (older templates have it; derived templates inherit it) the scheduler skips the resource check for actors that declare no size; the only bound left is the worker's --max-actors, default 1000
A WorkerPool without limits on the worker container the worker reads its capacity through the downward API, which returns the node's allocatable; every worker on the node registers the whole node
Expected Behavior

A template without limits is refused at create and derive time, or receives a documented default. A WorkerPool without limits is refused, or the controller sets worker limits from the pool spec. The registered capacity is logged at worker registration. The API guide says what happens when either is missing.

Steps to Reproduce

My cluster ran 890 and then 512 concurrent actors on 320 workers for a week with admission never engaging. Each of the eight workers on an n2-standard-8 had registered 7.5 CPU, 28.9 GiB and 1,000 actors as its own capacity. The API guide describes a per-worker budget that declared sizes are booked against, and that is what I assumed was happening.

Suggestion
  • Refuse a template without limits, or apply a documented default.
  • Refuse a WorkerPool without worker limits, or derive them.
  • Log the capacity a worker registers.
Sandbox Runtime

gVisor (runsc)

Agent Substrate Version / Commit SHA

0fdb67b5

Kubernetes Version & Environment

GKE 1.36.4-gke.2046002, Container-Optimized OS, us-central1

Host OS & Architecture

Linux x86_64

Relevant Logs and Diagnostic Output
# a derived template
resources:
  limits: {}

# a worker on an n2-standard-8 (8 workers on the node, each reports this)
status:
  capacity:
    actors: 1000
    resources:
      limits:
      - name: cpu
        quantity: 7510m
      - name: memory
        quantity: 28927020Ki
  state: WORKER_STATE_ACTIVE
Additional Context

Setup: two GKE worker pools. Pool A: 40 x n2-standard-8 (8 vCPU, 32 GiB, Intel Cascade Lake), 192 GB pd-balanced boot disk, 320 worker pods (8 per node). Pool B: 783 x e2-standard-8 (8 vCPU, 32 GiB, AMD Rome), 500 GB pd-balanced boot disk, 2,000 worker pods. ateapi at 2 to 12 replicas; Postgres on one n2-standard-16 with pd-ssd. Images: SWE-bench Verified (445 images, about 1.1 GB compressed median, 13 layers), R2E-Gym-V1 (8,101 images, 0.57 GB compressed median, 1.9 GB max, 14 layers, almost no layer sharing between images), and a 36,871-image SWE-RL set (0.5 to 1.4 GB compressed median per subset, up to 13 GB squashed single-layer images); unpacked size is 2 to 2.5 times compressed. All images pulled from an Artifact Registry repository in the same region with plain containerd pulls. Load: create N actors from per-image ActorTemplates with golden snapshots enabled, send one command each, delete; or keep a fleet and suspend/resume it per step. Observed on 0fdb67b5; the code involved is unchanged on main at a508f8fe.

Confirmation
  • I have searched existing issues and verified that this is not a duplicate.
  • I have verified that this issue occurs on the latest commit on main.
Dominant language
Go
Stars
5.9k
Forks
637
Avg merge
2d 1h
Merged PRs (30d)
523

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from agent-substrate/substrate

All issues in agent-substrate/substrate

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.