Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Let a one-shot request skip the prompt-cache work (~45 ms per call that is never reused)

Open
  • Is something wrong?
#830 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
52/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
cpp, python
Domain
ai-infra-agents, backend-api-design, performance

Research direction

Start by reading the prompt-cache flow in src/program/generate.cpp and how requests are handled in serve/server.py; the issue notes that serve/server.py does not currently read cache_prompt. Determine how a request-level opt-out should avoid the turn split and checkpoint without disabling cache reuse for other conversations. Done means one-shot requests can opt out, while interactive chats retain reuse; the CPU fallback and trace-counter questions are separate investigations.

Written by the indexing model from the issue text.

Description

A follow-up to part 2 of #519. Your explanation there of the fixed cost of the batched read (with about 40 % of the experts outside VRAM, a short batch still has to bring in nearly all the missing ones) matches what I see on 0.1.39: over 35 warm batched reads of 212 to 558 tokens the time fits about 245 ms + 0.33 ms per token. Following your suggestion, I moved the checkpoint to the end of the fixed instruction with --prompt-cache-root 64, which saves 20 to 40 ms per call. What is left on the cache side still costs about 45 ms per call.

Setup

  • RTX 5090 32 GB (no display), Ryzen 9 5950X (AVX2), 96 GB DDR4-3200, Windows 11
  • Strata 0.1.39 release, Swift IQ3_XXS, --expert-cache auto (14,864 slots, 24.2 GiB), --prefill auto:32768, --max-context 32768, --kv int8, --pcie-frac 0.20, --spec 4 --spec-min-p 0.70, --prompt-cache-root 256 for the numbers below (production now runs 64), STRATA_IQ_MT_MIN=1
  • The workload: a fixed system prompt of ~100 tokens, then a user message of 150 to 450 tokens of code, max_tokens 1, thinking off. Every call has a different user message, and the same server also takes interactive chats that do reuse their conversations.
  • The numbers below use the same shape built from public text: a 278-character system prompt (about 70 tokens) and excerpts of v0.1.32's src/program/generate.cpp, the same prompts in every run, 12 calls per size with the first 2 skipped, client on the same PC. I can share the script.

What the prompt cache costs on these calls

User message Prompt tokens read prompt_ms, cache on prompt_ms, --prompt-cache 0
~150 tokens 230 375 328
~300 tokens 379 434 388
~450 tokens 487 460 407

That is about 46 ms of prompt_ms and about 40 ms of wall time per call. With the cache on, the prompt is read in two parts, split at the last <|im_start|>, and a checkpoint is taken there. That checkpoint ends after the user message, which is different on every call, so nothing reuses it. With STRATA_TRACE=1, a 486-token prompt shows read 479 tokens (batched) in 403.5 ms, refilled 190 slots ... in 14.0 ms and read 6 tokens (windows) in 14.4 ms against prompt 486 tokens ... read in 460 ms. The trace leaves about 27 ms outside the timed reads, so the checkpoint itself costs at most that and the rest comes from the split.

--prompt-cache 0 removes this for the whole server, and the interactive chats lose their reuse with it. serve/server.py doesn't read cache_prompt, so a request can't ask for it.

Question

Could a request opt out of both the split at the last turn and the checkpoint when it has nothing to keep, for example by honouring cache_prompt: false, or automatically when a one-turn prompt's only reusable part is already a root? I'd expect it to save up to what --prompt-cache 0 saves on calls like these.

Separately: for a short batch where most of the missing experts are needed anyway, would running the non-resident ones on the CPU, as the decode windows do with misses, ever beat streaming them? If there's a trace counter that splits the batched read into transfer and compute time, I'd run it here.

I can share the traces and the script, and run a build with a change on this machine.

Dominant language
C++
Stars
11.6k
Forks
1k
Avg merge
7h 46m
Merged PRs (30d)
30

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from Niko1221/Strata

All issues in Niko1221/Strata

Similar issues

More C++ issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.