Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

Requests stall and time out above `--max-running-requests 8`; GPU stays at ~28% of TDP and only one CPU thread is busy

Aperta
  • Qualcosa non va?
#661 2 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
35/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
python
Ambito
ai-infra-agents, backend, performance

Direzione di ricerca

Start by reproducing the reported slowdown with the ft serve command and the fixed 200-item GSM8K workload at concurrency 4, 8, and 16; capture server batch logs and CPU and GPU counters as described. Read the existing reports #544 and #501 for related behavior. Identify where request scheduling or per-request timing diverges as concurrency rises; done means the cause is isolated and the workload no longer accumulates timeouts under the same conditions.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Summary

Serving Qwen3.8-Flash-Next (NVFP4) on 2× RTX 4080 SUPER with ft serve, increasing client concurrency from 4 → 8 → 16 makes things progressively worse, and at 16 the run collapses into a retry loop and never finishes.

At every concurrency level the GPU sits at ~91 W (of 320 W TDP), the KV cache uses ≤16%, and the mamba slot pool is never full — yet the client times out on a growing fraction of requests:

client concurrency wall time (200 GSM8K items) client timeouts #queue-req mamba slots KV usage GPU power
4 3432 s (57.2 min), finished 200/200 2 (0.01/task) 0 throughout 16/96 4–6% 91–93 W
8 4298 s (71.6 min), finished 192/200 20 (0.10/task) 0 throughout 32/96 7–10% 91–93 W
16 aborted at 134/200 after 51 min 111 (0.83/task) 3 → 14 and climbing 64/96 15–16% 91 W
24 server refuses to start (see below) — — — — —
32 server refuses to start (see below) — — — — —

The timeout rate grows roughly 10× per doubling of concurrency above 4, and the backlog (#queue-req) only appears once the timeouts start. It looks like a feedback loop: the scheduler cannot finish a request before the client gives up → the client retries → the retry occupies the server → the backlog grows → more timeouts.

Environment

  • ASUS ProArt Z890-CREATOR WIFI, Intel Core Ultra 9 285, BIOS 3401
  • 192 GB DDR5-6000, Ubuntu 26.04.1 LTS, kernel 7.0.0-38-generic
  • 2× RTX 4080 SUPER (32 GB each), model served on a single GPU
  • Model: Qwen3.8-Flash-Next NVFP4 checkpoint (~126 GB repo, experts in host RAM)
  • Engine: ft serve --model-path <nvfp4> --kv-reserve-tokens 131072 --max-running-requests N
    • --moe-cache-auto resolves to moe_cache_size=1927 num_pages=2048
    • mamba slot pool: 96, i.e. 4 slots per request

Method

lm-evaluation-harness, GSM8K, the same first 200 items in every run, 5-shot, thinking disabled, greedy. Client concurrency via NUM_CONCURRENT. The server was restarted cleanly (verified: GPU memory.used back under 500 MiB before starting) for every configuration, and each run was started only after the log printed API server is ready to serve.

Hardware and server counters were sampled every 2 s from nvidia-smi and the batch lines in the server log.

Evidence that the hardware is idle

Every sample, at every concurrency level:

GPU1: 100% util, ~91 W, 31 GB used   (320 W TDP)
token usage: 0.04 → 0.16             (KV cache, 4% → 16%)
#mamba-slot: 16/96 / 32/96 / 64/96   (never full)

100% utilization here only means "a kernel is running", not "the ALU is saturated" — the board draws 28% of its power budget.

Evidence that a single CPU thread is the limit

With 24 cores and load average ~1.1:

$ top -H -b -n2 -p <worker pid>
    PID USER  PR  NI    VIRT    RES  S  %CPU  COMMAND
 589808 root  20   0   ...   66.8g  R  99.0  python      <- one thread at ~99%

The worker process total is ~100% (i.e. exactly one core); other threads stay at 0.0%.

Fresh-engine drift (may be related to #544)

On a freshly started engine, fixed workload, fixed context (~20 K tokens, KV usage rising only 15 → 16%), the per-task time still creeps up:

21.0 → 21.6 → 21.6 → 22.2 → 22.9 s/task   over the first ~70 min
#queue-req average: 1.5 → 1.7 → 1.7 → 2.1 → 2.5

Since the context length does not grow, this does not look like the usual "longer context is slower" effect.

Capacity wall at higher concurrency (possibly a separate issue)

--max-running-requests 24 and 32 fail at startup while loading the model:

# N=32
AssertionError: cache budget too small: minimum plan (moe=1024 slots, kv=3200 pages)
needs 8031928320 B > budget -2434735829 B

# N=24
AssertionError: cache budget too small: minimum plan (moe=1024 slots, kv=2048 pages)
needs 6162481152 B > budget 3116221099 B

Lowering --kv-reserve-tokens shrinks the plan (3200 → 2048 pages, 8.03 → 6.16 GB) but the budget stays negative, so the shortfall is not the KV pool. Raising memory_ratio to 0.97 only recovers ~2.3 GB. This may simply be the linear-state (mamba slot) cache not fitting in 32 GB — reporting it here mainly to bound the usable concurrency range.

Relation to existing reports

  • #544 (closed) describes decode getting slower the longer a process runs. Our data shows the same direction with a fixed short context, which is why we are reporting it separately.
  • #501 describes cached-prefix prefill being much slower. Our requests are short (~2 K prompt tokens) and we still hit timeouts, so the two may share a root cause in the scheduler loop.

Reproduction

Server:

ft serve --model-path /path/to/nvfp4 --served-model-name nvfp4 \
  --host 127.0.0.1 --port 8080 --gpu <uuid> \
  --kv-reserve-tokens 131072 --max-running-requests 16

Client (lm-eval-harness, GSM8K, first 200 items):

HF_DATASETS_OFFLINE=1 NUM_CONCURRENT=16 EVAL_LIMIT=200 \
  python run_eval3.py <tag> gsm8k

What we can measure next

Happy to run anything you need: longer soak with fixed concurrency, per-request server timing, different --moe-cpu-threads, or a TP/2-GPU comparison. We have the machine available and can reproduce on demand.

Lingua principale
Python
Stelle
14.2k
Fork
1.4k
Merge medio
21h 39m
PR unite (30g)
100

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di FlashML-org/FreeToken

Tutte le issue di FlashML-org/FreeToken

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.