"parallel": 2 lowers total throughput on a single 48 GB card (100% expert-cache hits): keep MTP drafts in batch windows?
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 48/100
- Issue type
- Bug
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- cpp
- Domain
- performance
Research direction
Start by tracing the batch-window scheduling and MTP draft handling in the inference engine; the issue does not name specific files or tests. Reproduce the reported single-card benchmark with --spec 4 --mtp and compare no parallel against parallel: 2 at the listed request counts. Done means the behavior is understood and any change preserves or improves total throughput in those cases.
Written by the indexing model from the issue text.
Description
Could batch windows keep MTP drafts? On one card where the experts fit, "parallel": 2 lowers total throughput, because each slot decodes one token per window.
v0.1.39, one RTX PRO 5000 48 GB, Xeon 6148 (AVX2 kernels), Qwen3.8-Flash-Next IQ3_S, --spec 4 --mtp, 262K ctx. Decode expert cache hits 100%. Rates are streamed tokens per second, total across all requests:
no parallel |
parallel: 2 |
|
|---|---|---|
| 1 request | 97 | 64 |
| 2 at once | 103 | 61 |
| 4 at once | 100 | 71 |
strata batch: 800 windows, avg 1.99 rows, 16.71 ms/window = run 15.88 (CUDA0 GPU-reach wait 11.34 + CPU experts 3.05) ...
Happy to test a build.
- Dominant language
- C++
- Stars
- 11.6k
- Forks
- 1k
- Avg merge
- 7h 46m
- Merged PRs (30d)
- 30
Getting set up
This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from Niko1221/Strata
-
ROCm on Windows ZIPOpen
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
Niko1221/Strata#738 · 1 comment · 1 reaction ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 67/100
Niko1221/Strata#694 · 2 comments · 4 reactions ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
Niko1221/Strata#691 · 2 comments ·
Maintainers usually reply within 1 day
Similar issues
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 73/100
EchoTools/nevr-runtime#116 ·
Maintainers usually reply within 1 day
-
code-quality libc++
Difficulty 1/5 Under an hour Newbie friendliness 82/100
llvm/llvm-project#229284 ·
Maintainers usually reply within 1 day
-
test-issue
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
llvm/offload-test-suite#1557 ·
Maintainers usually reply within 1 day
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Maintainers usually reply within 1 day
-
iOS: hidden scale bar invalidates its intrinsic content size on every layout pass of MLNMapViewOpen
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
maplibre/maplibre-native#4723 ·
Maintainers usually reply within 1 day