Requests stall and time out above `--max-running-requests 8`; GPU stays at ~28% of TDP and only one CPU thread is busy

Start by reproducing the reported slowdown with the ft serve command and the fixed 200-item GSM8K workload at concurrency 4, 8, and 16; capture server batch logs and CPU and GPU counters as described. Read the existing reports #544 and #501 for related behavior. Identify where request scheduling or per-request timing diverges as concurrency rises; done means the cause is isolated and the workload no longer accumulates timeouts under the same conditions.

Summary

Serving Qwen3.8-Flash-Next (NVFP4) on 2× RTX 4080 SUPER with ft serve, increasing client concurrency from 4 → 8 → 16 makes things progressively worse, and at 16 the run collapses into a retry loop and never finishes.

At every concurrency level the GPU sits at ~91 W (of 320 W TDP), the KV cache uses ≤16%, and the mamba slot pool is never full — yet the client times out on a growing fraction of requests:

client concurrency	wall time (200 GSM8K items)	client timeouts	#queue-req	mamba slots	KV usage	GPU power
4	3432 s (57.2 min), finished 200/200	2 (0.01/task)	0 throughout	16/96	4–6%	91–93 W
8	4298 s (71.6 min), finished 192/200	20 (0.10/task)	0 throughout	32/96	7–10%	91–93 W
16	aborted at 134/200 after 51 min	111 (0.83/task)	3 → 14 and climbing	64/96	15–16%	91 W
24	server refuses to start (see below)	—	—	—	—	—
32	server refuses to start (see below)	—	—	—	—	—

The timeout rate grows roughly 10× per doubling of concurrency above 4, and the backlog (#queue-req) only appears once the timeouts start. It looks like a feedback loop: the scheduler cannot finish a request before the client gives up → the client retries → the retry occupies the server → the backlog grows → more timeouts.

Environment

• ASUS ProArt Z890-CREATOR WIFI, Intel Core Ultra 9 285, BIOS 3401
• 192 GB DDR5-6000, Ubuntu 26.04.1 LTS, kernel 7.0.0-38-generic
• 2× RTX 4080 SUPER (32 GB each), model served on a single GPU
• Model: Qwen3.8-Flash-Next NVFP4 checkpoint (~126 GB repo, experts in host RAM)
• Engine: ft serve --model-path <nvfp4> --kv-reserve-tokens 131072 --max-running-requests N
  • --moe-cache-auto resolves to moe_cache_size=1927 num_pages=2048
  • mamba slot pool: 96, i.e. 4 slots per request

Method

lm-evaluation-harness, GSM8K, the same first 200 items in every run, 5-shot, thinking disabled, greedy. Client concurrency via NUM_CONCURRENT. The server was restarted cleanly (verified: GPU memory.used back under 500 MiB before starting) for every configuration, and each run was started only after the log printed API server is ready to serve.

Hardware and server counters were sampled every 2 s from nvidia-smi and the batch lines in the server log.

Evidence that the hardware is idle

Every sample, at every concurrency level:

GPU1: 100% util, ~91 W, 31 GB used   (320 W TDP)
token usage: 0.04 → 0.16             (KV cache, 4% → 16%)
#mamba-slot: 16/96 / 32/96 / 64/96   (never full)

100% utilization here only means "a kernel is running", not "the ALU is saturated" — the board draws 28% of its power budget.

Evidence that a single CPU thread is the limit

With 24 cores and load average ~1.1:

$ top -H -b -n2 -p <worker pid>
    PID USER  PR  NI    VIRT    RES  S  %CPU  COMMAND
 589808 root  20   0   ...   66.8g  R  99.0  python      <- one thread at ~99%

The worker process total is ~100% (i.e. exactly one core); other threads stay at 0.0%.

Fresh-engine drift (may be related to #544)

On a freshly started engine, fixed workload, fixed context (~20 K tokens, KV usage rising only 15 → 16%), the per-task time still creeps up:

21.0 → 21.6 → 21.6 → 22.2 → 22.9 s/task   over the first ~70 min
#queue-req average: 1.5 → 1.7 → 1.7 → 2.1 → 2.5

Since the context length does not grow, this does not look like the usual "longer context is slower" effect.

Capacity wall at higher concurrency (possibly a separate issue)

--max-running-requests 24 and 32 fail at startup while loading the model:

# N=32
AssertionError: cache budget too small: minimum plan (moe=1024 slots, kv=3200 pages)
needs 8031928320 B > budget -2434735829 B

# N=24
AssertionError: cache budget too small: minimum plan (moe=1024 slots, kv=2048 pages)
needs 6162481152 B > budget 3116221099 B

Lowering --kv-reserve-tokens shrinks the plan (3200 → 2048 pages, 8.03 → 6.16 GB) but the budget stays negative, so the shortfall is not the KV pool. Raising memory_ratio to 0.97 only recovers ~2.3 GB. This may simply be the linear-state (mamba slot) cache not fitting in 32 GB — reporting it here mainly to bound the usable concurrency range.

Relation to existing reports

• #544 (closed) describes decode getting slower the longer a process runs. Our data shows the same direction with a fixed short context, which is why we are reporting it separately.
• #501 describes cached-prefix prefill being much slower. Our requests are short (~2 K prompt tokens) and we still hit timeouts, so the two may share a root cause in the scheduler loop.

Reproduction

Server:

ft serve --model-path /path/to/nvfp4 --served-model-name nvfp4 \
  --host 127.0.0.1 --port 8080 --gpu <uuid> \
  --kv-reserve-tokens 131072 --max-running-requests 16

Client (lm-eval-harness, GSM8K, first 200 items):

HF_DATASETS_OFFLINE=1 NUM_CONCURRENT=16 EVAL_LIMIT=200 \
  python run_eval3.py <tag> gsm8k

What we can measure next

Happy to run anything you need: longer soak with fixed concurrency, per-request server timing, different --moe-cpu-threads, or a TP/2-GPU comparison. We have the machine available and can reproduce on demand.