serve: a >100k-token prompt deadlocks the pipelined verify-window reader (--pipeline-windows >= 1) - the #29 watchdog aborts the engine and the server reloads the model

Chiusa
  • Qualcosa non va?
#1,341 3 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 4 giorni

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
45/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
cpp
Ambito
backend, performance

Direzione di ricerca

Start with src/program/generate.cpp, especially read_windows_pl around lines 8521–8522 and the selection condition around line 8608; compare its verify-window handshake with the non-pipelined reader. Use a cold prompt over 100k tokens with --pipeline-windows >= 1 to reproduce, and inspect the attached stall dumps if available. Done means long cold reads complete without hanging or triggering the watchdog, while the --pipeline-windows 0 workaround remains unnecessary.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Summary

On engine 0.1.40 (server 0.1.40.1) with a dual-GPU layer split, a prompt of about 120k tokens hangs while "reading the prompt". The engine's own #29 watchdog fires after 60 s, writes a stall dump, and aborts the engine; the server then reloads the model (about 2 minutes) and the request fails. Setting --pipeline-windows 0 makes the same long cold reads pass.

The hang is in the pipelined verify-window prompt reader (read_windows_pl), which is only used when --pipeline-windows >= 1. The engine prints no illegal memory access in this reproduction.

Environment

  • engine 0.1.40, server 0.1.40.1 (82f46a8), Windows 11, CUDA 13.0 release build, compute capability 8.6
  • 2x RTX Laptop (3080 + 3070), 16384 MiB each, driver 616.56; CPU Intel Xeon E5-2680 v4
  • model Qwen3.8-Flash-Next IQ3_XXS (24576 experts, 48 layers)
  • args: --pack ... --native ... --ple-gguf ... --expert-profile ... --expert-cache auto --prefill auto:32768 --spec 4 --spec-min-p 0.70 --mtp ... --max-context 262144 --kv int8 --kv-resident 32768 --vision --vram-reserve-mib 690 --pool-workers 8 --trim-stage-weights --pipeline-windows 2 --layer-split 26
  • 630 MiB of VRAM free with everything loaded; about 34 GB of RAM free -> not an OOM.

Observed

Production, same config: a request with a 129,038-token prompt failed (finish=error, 0 output tokens) and the server reloaded the engine. Earlier the same day a 107,882-token prompt ended with prefill copy_i32: an illegal memory access was encountered (the sticky-label class) plus the #29 watchdog, in the batched reader (stage "reading the prompt (batched): layer 27 of the prompt chunk from token 107882").

Reproduced 2/2 with a cold ~120,000-token prompt (a fresh, uncached prompt; nothing reused):

strata serve: no progress for 60 s during a request (reading the prompt (pipelined verify windows), from token 120013) - stopping the engine so the server starts it again (issue #29)
strata serve: stall report (engine 0.1.40): stage "reading the prompt (pipelined verify windows), from token 120013" for 57 s; 0 layers served since the last finished step (0 = stopped, more = slow)
  expert pool: epoch 0, batch epoch 0: 0 of 0 jobs claimed, 0 done; 8 of 8 workers parked, 8 sleeping; mode 0
  wrote the thread stacks to C:\strata\Strata\strata-stall-16932.dmp (attach it to the issue)
strata: released the verify window's GPU waits (#267): the GPU finished in 0 ms
strata: released the verify window's GPU waits (#267): the GPU finished in 0 ms
strata: released the verify window's GPU waits (#267): the GPU finished in 0 ms
strata: released the verify window's GPU waits (#267): the GPU finished in 0 ms
strata: released the verify window's GPU waits (#267): the GPU did not finish within 5 s
strata: released the verify window's GPU waits (#267): the GPU did not finish within 5 s
strata: released the verify window's GPU waits (#267): the GPU finished in 20 ms
strata: released the verify window's GPU waits (#267): the GPU finished in 20 ms

The second run was identical (for 56 s, dump strata-stall-22140.dmp). The prompt read rate was normal before the hang (120k tokens in about 95 s, ~1260 tok/s), so it is not a slow-read timeout.

Note the expert pool line: 0 of 0 jobs claimed -> no prefill job is running at all; the reader is parked waiting on something that is never signalled, not merely slow.

Instrumentation: the two reproduction runs set CUDA_LAUNCH_BLOCKING=1 and STRATA_PF_STEP_SYNC=1 to localise the stage. The production failures ran with the same config and no instrumentation. Launch blocking normally makes races less likely, which points to a deterministic missing-signal deadlock rather than a timing race.

Suspected cause

The stage label reading the prompt (pipelined verify windows) is emitted only from read_windows_pl (src/program/generate.cpp:8521; the label is at :8522). That reader is selected only when --pipeline-windows >= 1:

// src/program/generate.cpp:8608
if (pipe && pl_pw >= 1 && std::getenv("STRATA_LOGPOS") == nullptr) return read_windows_pl(a, b, e);

with int pl_pw = pipe ? o.pipeline_windows : 0; (generate.cpp:8494). So the deadlock is in the overlapped cross-stage verify-window prompt read. The engine does not report an illegal access in this reproduction, so the prefill copy_i32 label seen in production is the sticky cudaGetLastError() mis-attribution (see src/kernels/cuda/qsa.cu:77) rather than the actual faulting kernel.

The earlier production signature (batched reader, layer 27) may be a sibling manifestation of the same verify-window handshake; I could not attribute it separately.

Workaround (verified)

--pipeline-windows 0: 3/3 cold long reads pass, no fault, no stall, no reload:

prompt tokens result time cached_tokens
121,027 HTTP 200 64.8 s 0
126,027 HTTP 200 68.7 s 0
131,026 HTTP 200 70.8 s 0

(all genuinely cold; --prefill auto:32768 unchanged). Before the change: 2/2 cold ~120k reads stalled.

Trade-off: this gives up the ~+16-21% decode and ~129 MiB of headroom that --pipeline-windows provides (docs/MULTI_GPU.md).

Dumps

strata-stall-16932.dmp, strata-stall-22140.dmp (this reproduction), strata-stall-23716.dmp (production, batched path, layer 27 / token 107882). Happy to attach any of them.

Related

  • #29 (engine watchdog), #267 (verify-window GPU-wait release), #964 (open: dual-GPU + >100k prompt + illegal access + verify-window hang), #1275.
Lingua principale
C++
Stelle
11.6k
Fork
1k
Nessuna PR unita negli ultimi 30g

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di Niko1221/Strata

Tutte le issue di Niko1221/Strata

Issue simili

Altre issue su C++

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.