DeepSeek V4.1 Flash TP3 on GB10: with four requests, the balanced policy verifies most rounds at width one; throughput is 8% faster on prose

Aperta
  • Qualcosa non va?
#152 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
38/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
rust
Ambito
backend, performance

Direzione di ricerca

Start with helios/src/scheduler/policy.rs, utility.rs, and learning.rs, which the issue identifies as the scheduler policy code; compare how the balanced and throughput weights affect verification width at batch four. The measurements show lower throughput with balanced settings, but inter-token latency was not measured, so collect that data and weigh the cadence trade-off before deciding whether the weights should change.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

area:scheduling model:deepseek-v4.1 performance triage:design-review

Setup

  • Three DGX Spark GB10 hosts, DeepSeek V4.1 Flash TP3, package exported with --max-concurrency 4 and --mtp-draft-tokens 5 (verification width 6), served with flare run at ca1f960e with default options apart from the flags below.
  • The scheduler policy code (helios/src/scheduler/policy.rs, utility.rs, learning.rs) is unchanged between ca1f960e and current main. On a main (921afd5c) package with --scheduler-policy throughput, the four-request aggregates were 89.4 / 92.2 tok/s on prose and 139.9 / 146.5 on code, in line with the table below.

Measurements

Streamed chat completions, temperature 0, thinking off, 512 output tokens per request. "Real text" is four prose and four code prompts; "random words" uses random-word prompts with a fixed seed. Aggregate tok/s with four concurrent requests:

balanced (default) --no-mtp-adaptive-verification --scheduler-policy throughput
Prose, real text 83.9 / 85.6 / 86.5 90.6 / 92.5 / 93.7 92.0 / 93.6
Code, real text 135.8 / 141.8 / 142.0 141.6 / 142.5 / 145.5 145.9 / 144.6
Random words 89.2 102.3 not measured
  • With one request the settings are within noise: prose 43-46 tok/s, code 76-87.
  • With four requests, per-request speed is also higher without the balanced pricing: on prose, 21.4-22.0 tok/s per stream with balanced, and 24.0-25.4 with the other two settings.

Rounds at batch 4 (flare_speculative_rounds_total):

width 1 width 6
balanced, one real-text run 416 165
throughput, two real-text runs 1 685
  • With balanced, most four-request rounds verify no drafts.
  • On random words at batch 4, a width-6 round emitted about 7.0 tokens (1.76 per request), against 4 for a width-1 round.

Question

throughput differs from balanced only by zeroing delivery_fairness (4.0) and decode_age_price (0.1). On this setup the balanced weights lead the scheduler to choose width-1 rounds at batch 4, which lowers both aggregate and mean per-request token rates. I did not measure inter-token latency, so I cannot say whether output cadence improves in exchange. Is this the intended trade-off for this case, or would it make sense to revisit the weights? I can run other weights, collect per-graph timings, or measure inter-token gaps.

Lingua principale
Rust
Stelle
800
Fork
106
Merge medio
1g 3h
PR unite (30g)
100

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di NVIDIA/hyperflare-inference

Tutte le issue di NVIDIA/hyperflare-inference

Issue simili

Altre issue su Rust

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.