Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

524K via rope-scaling on consumer Blackwell (sm_120, no clusters): the histogram fallback works end to end — field data + two budget gotchas on 16GB cards

Open
  • Is something wrong?
#760 2 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
54/100
Issue type
Documentation
Clarity
Mostly clear
Activity status
Active
Tech stack
cpp
Domain
documentation, performance

Research direction

Start with the README section describing 384K/512K experimental rope scaling and check how it explains hardware support and fallback behavior. Use the reported sm_120 run as evidence for a concise note that selection uses the histogram fallback on consumer Blackwell. Done when the guidance sets that expectation without suggesting the fallback is broken.

Written by the indexing model from the issue text.

Description

Your README lists 384K/512K (experimental) via the rope-scaling CLI. Just filed field data for the corner nobody can test in CI: consumer Blackwell (RTX 5060 Ti 16GB, sm_120) has no thread-block clusters, so qsa_block_topk runs entirely through the histogram fallback at those contexts — and it works, end to end, past the trained window.

Rig: 4x 5060 Ti (gen5, x8/x4/x8/x4), 9950X, 128GB DDR5-5000. Engine: the eddoursul/Strata custom fork extended to 4-GPU expert tiers (our build: https://github.com/johnmosesventura06/strata-4gpu, PR https://github.com/eddoursul/Strata/pull/4 carries your qsa_select.cu + rope_scaling machinery onto that fork), UD-Q4_K_XL, int8 KV, MTP on.

At --rope-scaling yarn --rope-scale 2.0 --max-context 524288:

  • boot READY, rope scaling yarn, factor 2 (freq_scale 0.5, mscale 1.069315) — mscale exactly 1 + 0.1*ln(2)
  • 281,535-token cold prompt: 1,418 t/s prefill; decode at 225K–312K depth: 84–111 t/s
  • warm replay of the 312K prompt: 12 ms (312,134 tokens from checkpoints)
  • identity gate: --rope-scale 1.0 @196,608 reproduced the unscaled decode band (98.2 t/s) — the scaling path is bit-neutral when off

Two budget facts on 16GB cards that the CLI docs don't mention:

  1. At 524K the KV pool (~3.7GB int8) lands before the auto expert cache sizes itself; main-card residency drains to ~202 slots and ranked experts fall to host/tier. Measured cost is modest (decode -5% per residency halving), but the staging consequence is hard: --prefill 24576's ~2.4GB of device buffers can no longer borrow or allocate and the engine dies between load and READY. --prefill 4096 boots; the chunk clamp itself costs 18% of prefill (A/B'd with residency held constant: 2,159 -> 1,778 t/s at 256K with a fat cache).
  2. --vram-reserve-mib is not the escape hatch: raising it to buy the staging back starves the ~1.4GB weight arena first and dies before that. The real lever at 524K is KV-side bytes (k8v4 — we haven't gaunted it yet).

And one doc suggestion: 384K/512K "experimental" probably needs a footnote that on sm_120 (all consumer Blackwell) selection lands on the histogram fallback rather than the cluster kernel — users will otherwise read "unsupported geometry or cap"-adjacent failures as broken rather than as the intended fallback... which, to be fair, this run proves is not broken.

Happy to re-run any arm on this box if it helps you pin the fallback's geometry limits.

Dominant language
C++
Stars
11.6k
Forks
1k
Avg merge
7h 46m
Merged PRs (30d)
30

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from Niko1221/Strata

All issues in Niko1221/Strata

Similar issues

More C++ issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.