[Benchmark] 0.1.56 to 0.1.58 decode flat on a 4060 Ti + 3060 layer split; the 12 GB fixes do not apply when a second card holds the experts

Open Beginner friendly
  • Is something wrong?
#5,501 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 4 days

Nobody has claimed this yet.

Assessment

Difficulty
1/5
Estimated time
Under an hour
Newbie friendliness
78/100
Issue type
Documentation
Clarity
Clearly specified
Activity status
Active
Tech stack
cpp
Domain
performance

Research direction

Start by reading the release notes for 0.2.11 and 0.2.12 and the behavior described for --remote-expert-opt in the issue. Add a one-line clarification that second-card expert placement bypasses both the file-cache fix and the CPU/upload share ceiling change. Done means the release notes make that limitation clear for split-card setups.

Written by the indexing model from the issue text.

Description

Setup. Linux x86 (CUDA 12 engine), RTX 4060 Ti 16 GB + RTX 3060 12 GB, layer split 28/20, --remote-expert-opt --pipeline-windows 2 --pool-workers 11 --spec 4 --spec-min-p 0.5 --kv int8, 262K context, Qwen3.8-Flash-Next IQ3_S (abliterated pack), one request at a time. Xeon W-2245 (8C16T). Fixed prompts (prose + code, 400 tokens out, temperature 0), 2 rounds per run, runs interleaved across versions by swapping only exe + LD_LIBRARY_PATH.

Result: 0.1.56 → 0.1.58 decode is flat on this rig.

engine run means (tok/s)
0.1.56 62.0, 64.6
0.1.58 63.3, 62.7, 62.6, 64.2

Draft acceptance 63-67% both sides, expert cache hit 83-91% both sides. Nothing regressed; nothing gained either.

Why the two headline 12 GB changes did not land here:

  1. The file-cache fix (0.2.12, "the second start-up check no longer counts the engine's own memory as free"): the switch line expert reads now go through the file cache never appears in our log on 0.1.58 (0 occurrences), and it never appeared on 0.1.56 either — with --remote-expert-opt the missed experts go to the second card (the 3060), not to the file cache, so the misclassification path the fix addresses is not the one our rig walks. The RTX 3060 3-of-3 → 0-of-3 improvement in the release notes is presumably the single-card placement.
  2. The CPU/upload share ceiling 0.60 → 0.95 (0.2.11): same reason — it prices the CPU lane for missed experts, and on this pair the 3060 absorbs them instead. The +13% figure was measured on two PRO 6000 cards held to 12 GB free each, where the CPU lane really is the overflow.

What we did keep 0.1.58 for: the tool-call fixes (#5312, #5409, #5276, #5281, #4828, #4832) matter for the agent traffic this server serves, and decode did not regress. So: flat speed, net positive on reliability.

Suggestion. The release notes' 12 GB framing reads as if every 3060-class rig gets these wins; a one-line note that --remote-expert-opt (or any second-card expert placement) bypasses both paths would save the next split-pair user the same A/B we just ran.

Dominant language
C++
Stars
11.6k
Forks
1k
No merged PRs in 30d

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from Niko1221/Strata

All issues in Niko1221/Strata

Similar issues

More C++ issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.