[Bug]: A template without resource limits is admitted with no resource check, and a worker without limits registers the node's allocatable as its capacity

Abierto
  • ¿Algo no está bien?
#3,479 3 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
35/100
Tipo de issue
Error
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
go, kubernetes
Área
backend-api-design, infrastructure

Línea de trabajo

Start by tracing template create/derive validation, WorkerPool worker-limit handling, scheduler admission checks, and worker capacity registration; the issue does not name specific files or tests. Review the API guide alongside those entry points to understand the documented per-worker budget. Done means missing limits are handled consistently, registered capacity is logged, and the API guide explains the behavior.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

area/api-machinery area/scheduling kind/bug
What happened?

Two defaults combine so that no admission control runs, with nothing reporting it.

Default Effect
A template with empty resources.limits is accepted (older templates have it; derived templates inherit it) the scheduler skips the resource check for actors that declare no size; the only bound left is the worker's --max-actors, default 1000
A WorkerPool without limits on the worker container the worker reads its capacity through the downward API, which returns the node's allocatable; every worker on the node registers the whole node
Expected Behavior

A template without limits is refused at create and derive time, or receives a documented default. A WorkerPool without limits is refused, or the controller sets worker limits from the pool spec. The registered capacity is logged at worker registration. The API guide says what happens when either is missing.

Steps to Reproduce

My cluster ran 890 and then 512 concurrent actors on 320 workers for a week with admission never engaging. Each of the eight workers on an n2-standard-8 had registered 7.5 CPU, 28.9 GiB and 1,000 actors as its own capacity. The API guide describes a per-worker budget that declared sizes are booked against, and that is what I assumed was happening.

Suggestion
  • Refuse a template without limits, or apply a documented default.
  • Refuse a WorkerPool without worker limits, or derive them.
  • Log the capacity a worker registers.
Sandbox Runtime

gVisor (runsc)

Agent Substrate Version / Commit SHA

0fdb67b5

Kubernetes Version & Environment

GKE 1.36.4-gke.2046002, Container-Optimized OS, us-central1

Host OS & Architecture

Linux x86_64

Relevant Logs and Diagnostic Output
# a derived template
resources:
  limits: {}

# a worker on an n2-standard-8 (8 workers on the node, each reports this)
status:
  capacity:
    actors: 1000
    resources:
      limits:
      - name: cpu
        quantity: 7510m
      - name: memory
        quantity: 28927020Ki
  state: WORKER_STATE_ACTIVE
Additional Context

Setup: two GKE worker pools. Pool A: 40 x n2-standard-8 (8 vCPU, 32 GiB, Intel Cascade Lake), 192 GB pd-balanced boot disk, 320 worker pods (8 per node). Pool B: 783 x e2-standard-8 (8 vCPU, 32 GiB, AMD Rome), 500 GB pd-balanced boot disk, 2,000 worker pods. ateapi at 2 to 12 replicas; Postgres on one n2-standard-16 with pd-ssd. Images: SWE-bench Verified (445 images, about 1.1 GB compressed median, 13 layers), R2E-Gym-V1 (8,101 images, 0.57 GB compressed median, 1.9 GB max, 14 layers, almost no layer sharing between images), and a 36,871-image SWE-RL set (0.5 to 1.4 GB compressed median per subset, up to 13 GB squashed single-layer images); unpacked size is 2 to 2.5 times compressed. All images pulled from an Artifact Registry repository in the same region with plain containerd pulls. Load: create N actors from per-image ActorTemplates with golden snapshots enabled, send one command each, delete; or keep a fleet and suspend/resume it per step. Observed on 0fdb67b5; the code involved is unchanged on main at a508f8fe.

Confirmation
  • I have searched existing issues and verified that this is not a duplicate.
  • I have verified that this issue occurs on the latest commit on main.
Lenguaje dominante
Go
Estrellas
5.9k
Forks
637
Merge medio
2 d 2 h
PR fusionados (30 d)
528

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de agent-substrate/substrate

Todos los issues de agent-substrate/substrate

Issues similares

Más issues de Go

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.