Worker aborts with V8 OOM in JSON.parse when several exec requests carry large stdin payloads; all leases lost

Aperta
  • Qualcosa non va?
#1,242 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
35/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
node.js, typescript
Ambito
api, backend

Direzione di ricerca

Start at the worker and gateway handling for POST /v1/leases/:id/exec, then trace where the request body is materialized and parsed. The issue does not name files or tests, so locate the relevant entry points and existing request-size or concurrency limits first. Done means oversized or concurrent large stdin requests are rejected with a typed error instead of crashing the worker; streaming or preserving leases across restarts are broader alternatives.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Summary

A simlock 0.5.0 worker (Node 22.23.4, macOS, gateway mode) aborts with a V8 out-of-memory error while parsing an incoming message with JSON.parse. This happens when several POST /v1/leases/<id>/exec requests carrying a large stdin (base64 payloads of tens of MB) are in flight at once. Every lease on that worker is lost: in-flight execs return 503 after about 105 s, and later calls get 404 UNKNOWN_LEASE.

Evidence

Crash report node-2026-11-03-144256.ips:

EXC_CRASH (SIGABRT) — abort() called
node::OOMErrorHandler
v8::internal::V8::FatalProcessOutOfMemory
v8::internal::Heap::CollectGarbage
v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath
v8::internal::FactoryBase::NewRawStringWithMap<SeqOneByteString>
v8::internal::JsonParser<unsigned char>::MakeString
v8::internal::JsonParser<unsigned char>::Parse
v8::internal::Builtin_JsonParse
  • Worker log: the next start at 13:42:56Z logs Recovered stale daemon socket / Daemon started and re-establishes the gateway uplink. launchctl reports the last exit as -6.
  • Gateway log for one lease: device.exec → POST …/exec 503 after 105,242 ms → the client releases → POST …/exec 404 UNKNOWN_LEASE.
  • Workload: a tester-army e2e suite with 10 workers. Each worker uploads an ~89 MB fixture archive (deflated, base64 in stdin) through exec (simctl spawn <helper> untar) at the same moment. The host's load average was about 70.
  • The same burst also produced Cloudflare 524s and connection resets on other runs, because each exec holds a single long request.

Root cause

The worker (or its gateway uplink) materialises each exec request, stdin included, as one JSON string and parses it on the heap. Concurrent large stdin payloads exceed the V8 heap limit, and the process aborts instead of rejecting the request.

Suggested fix

  1. Bound the accepted exec stdin size per request, and the total in flight per worker. Reject an over-limit request with a typed error (413/PAYLOAD_TOO_LARGE) instead of crashing.
  2. Stream stdin through the uplink and into the child process instead of parsing it as a single JSON string, or offer a separate upload endpoint, so large file transfers don't sit on the JSON path.
  3. Optionally, keep lease state across a worker restart (the leases were lost even though the devices survived).

On the client side, we are capping concurrent large uploads (consumer issue linked below), but a worker should not be crashable by request size.

Lingua principale
TypeScript
Stelle
535
Fork
21
Merge medio
9h 53m
PR unite (30g)
435

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di callstackincubator/simlock

Tutte le issue di callstackincubator/simlock

Issue simili

Altre issue su TypeScript

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.