Attempt containers on usine-worker-1 hit the memory limit on 8 attempts in 7 days: record what was killed and surface it

Aperta
  • Qualcosa non va?
#1,041 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
38/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
docker, github, node.js, typescript
Ambito
devops, observability, testing

Direzione di ricerca

Start by tracing how container memory kills are recorded when an attempt is removed, then follow how attempt details and statistics are rendered. Check the worker guide and the repository gate commands, including pnpm test, Turbo concurrency, and Vitest maxWorkers. Done means attempt pages show kill counts and commands, statistics include repeated kills, and the full gate's memory behavior meets the stated acceptance criteria.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

The orchestrator logs "The kernel killed processes in the container for memory" at container removal for a steady share of developer attempts on usine-worker-1. Over the 7 days to 2026-11-07, 8 attempts recorded kills (1 to 4 each, memoryMib: 6656), for example 9aff16c830153977f2acb767f32a01d1/implement/1 (#1024, 2 kills) and 2c4bd9db14c874ef1b015a52d2e7de14/implement/1 (4 kills). Every one of those attempts still completed, so the kills hit test or build subprocesses rather than the harness, and the agent most likely saw a crashed vitest worker or a failed pnpm build task and retried or moved on without saying so.

What is missing:

  1. The record counts kills but not what died. Capture the killed process's command line (from the cgroup memory.events plus dmesg/journalctl -k on the worker host at removal time, or from docker events --filter event=oom while the container runs) and store it with the attempt, so the operator can tell a vitest worker from the harness itself.
  2. Surface it. An attempt with memory kills shows nothing on the task page or the attempt page; the only trace is the WARN in the journal. Show the count and the killed commands on the attempt page, and count repeated kills in the statistics page as a quality signal next to first time right.
  3. Sizing. 6656 MiB is the per-attempt limit on usine-worker-1 (1 slot). The repository gate runs pnpm test across packages in parallel; the worker guide should state the memory the gate needs, and the limit or the gate's parallelism (--concurrency for turbo, maxWorkers for vitest) should fit it. Decide which one to change after item 1 shows what dies.

Acceptance: the attempt page lists the killed commands for an attempt with kills; a developer attempt running the full gate on usine-worker-1 completes without kills, or the kills name a process the gate is expected to shed.

Lingua principale
TypeScript
Stelle
1
Fork
0
Merge medio
3h 47m
PR unite (30g)
388

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di basaltbytes/usine

Tutte le issue di basaltbytes/usine

Issue simili

Altre issue su TypeScript

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.