Make golden snapshot bake concurrency configurable, and tell the caller whether an actor restored from a golden or booted from the image

Abierto
  • ¿Algo no está bien?
#3,484 1 comentario 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
46/100
Tipo de issue
Nueva funcionalidad
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
gcp, go, kubernetes
Área
backend, observability

Línea de trabajo

Start by tracing the template reconciler's golden-bake worker pool and the actor startup path; look for existing configuration flags and metrics to follow. The issue names no files or tests, so locate the relevant entry points before estimating the change. Done means bake concurrency is configurable, callers can tell whether an actor restored from a golden or booted from an image, and the bake backlog is exposed as a metric.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

area/api-machinery area/lifecycle kind/feature
What happens today

The template reconciler bakes goldens with a fixed pool of five workers per ateapi replica; a bake is about two minutes (boot, readiness wait, upload). Template creation has no such limit, so a job that derives thousands of templates quickly leaves the bakes far behind. While a template has no golden, its actors boot from the image, and nothing on the actor says so.

Templates created Time to create Backlog
8,101 9 min 45 min later most had no golden; every actor had booted
36,000 45 min about 20 h on 2 replicas, about 7 h on 12

Scaling ateapi to twelve replicas was the only lever, which ties the bake rate to API replica count by accident.

Proposal
  • Make the bake concurrency a flag.
  • Report on the actor whether it started from a golden or booted from the image.
  • Expose the bake backlog as a metric, so a caller can wait for it.
How to see it
  1. Create a few thousand ActorTemplates quickly, one per image, golden snapshots enabled.
  2. kubectl-ate get actor-templates -a ATESPACE; count empty golden tags over time. Create actors meanwhile: none produce a restore record.
  3. Scale ateapi; the rate changes proportionally.
Setup

Setup: two GKE worker pools. Pool A: 40 x n2-standard-8 (8 vCPU, 32 GiB, Intel Cascade Lake), 192 GB pd-balanced boot disk, 320 worker pods. Pool B: 783 x e2-standard-8 (8 vCPU, 32 GiB, AMD Rome), 500 GB pd-balanced boot disk, 2,000 worker pods. ateapi at 2 to 12 replicas; Postgres on one n2-standard-16 with pd-ssd. Images: SWE-bench Verified (445, about 1.1 GB compressed median, 13 layers), R2E-Gym-V1 (8,101, 0.57 GB compressed median, 1.9 GB max, 14 layers, almost no layer sharing), a 36,871-image SWE-RL set (0.5 to 1.4 GB compressed median per subset, up to 13 GB squashed single-layer images); unpacked size 2 to 2.5 times compressed; plain containerd pulls from a same-region Artifact Registry. gVisor (runsc), GKE 1.36.4-gke.2046002, Container-Optimized OS, Linux x86_64. Observed on Substrate 0fdb67b5; the code cited is unchanged on main at a508f8fe.

Lenguaje dominante
Go
Estrellas
6.1k
Forks
676
Merge medio
2 d 1 h
PR fusionados (30 d)
481

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de agent-substrate/substrate

Todos los issues de agent-substrate/substrate

Issues similares

Más issues de Go

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.