Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

Let a one-shot request skip the prompt-cache work (~45 ms per call that is never reused)

Chiusa
  • Qualcosa non va?
#830 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
52/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
cpp, python
Ambito
ai-infra-agents, backend-api-design, performance

Direzione di ricerca

Start by reading the prompt-cache flow in src/program/generate.cpp and how requests are handled in serve/server.py; the issue notes that serve/server.py does not currently read cache_prompt. Determine how a request-level opt-out should avoid the turn split and checkpoint without disabling cache reuse for other conversations. Done means one-shot requests can opt out, while interactive chats retain reuse; the CPU fallback and trace-counter questions are separate investigations.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

A follow-up to part 2 of #519. Your explanation there of the fixed cost of the batched read (with about 40 % of the experts outside VRAM, a short batch still has to bring in nearly all the missing ones) matches what I see on 0.1.39: over 35 warm batched reads of 212 to 558 tokens the time fits about 245 ms + 0.33 ms per token. Following your suggestion, I moved the checkpoint to the end of the fixed instruction with --prompt-cache-root 64, which saves 20 to 40 ms per call. What is left on the cache side still costs about 45 ms per call.

Setup

  • RTX 5090 32 GB (no display), Ryzen 9 5950X (AVX2), 96 GB DDR4-3200, Windows 11
  • Strata 0.1.39 release, Swift IQ3_XXS, --expert-cache auto (14,864 slots, 24.2 GiB), --prefill auto:32768, --max-context 32768, --kv int8, --pcie-frac 0.20, --spec 4 --spec-min-p 0.70, --prompt-cache-root 256 for the numbers below (production now runs 64), STRATA_IQ_MT_MIN=1
  • The workload: a fixed system prompt of ~100 tokens, then a user message of 150 to 450 tokens of code, max_tokens 1, thinking off. Every call has a different user message, and the same server also takes interactive chats that do reuse their conversations.
  • The numbers below use the same shape built from public text: a 278-character system prompt (about 70 tokens) and excerpts of v0.1.32's src/program/generate.cpp, the same prompts in every run, 12 calls per size with the first 2 skipped, client on the same PC. I can share the script.

What the prompt cache costs on these calls

User message Prompt tokens read prompt_ms, cache on prompt_ms, --prompt-cache 0
~150 tokens 230 375 328
~300 tokens 379 434 388
~450 tokens 487 460 407

That is about 46 ms of prompt_ms and about 40 ms of wall time per call. With the cache on, the prompt is read in two parts, split at the last <|im_start|>, and a checkpoint is taken there. That checkpoint ends after the user message, which is different on every call, so nothing reuses it. With STRATA_TRACE=1, a 486-token prompt shows read 479 tokens (batched) in 403.5 ms, refilled 190 slots ... in 14.0 ms and read 6 tokens (windows) in 14.4 ms against prompt 486 tokens ... read in 460 ms. The trace leaves about 27 ms outside the timed reads, so the checkpoint itself costs at most that and the rest comes from the split.

--prompt-cache 0 removes this for the whole server, and the interactive chats lose their reuse with it. serve/server.py doesn't read cache_prompt, so a request can't ask for it.

Question

Could a request opt out of both the split at the last turn and the checkpoint when it has nothing to keep, for example by honouring cache_prompt: false, or automatically when a one-turn prompt's only reusable part is already a root? I'd expect it to save up to what --prompt-cache 0 saves on calls like these.

Separately: for a short batch where most of the missing experts are needed anyway, would running the non-resident ones on the CPU, as the decode windows do with misses, ever beat streaming them? If there's a trace counter that splits the batched read into transfer and compute time, I'd run it here.

I can share the traces and the script, and run a build with a change on this machine.

Lingua principale
C++
Stelle
11.6k
Fork
1k
Merge medio
7h 46m
PR unite (30g)
30

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di Niko1221/Strata

Tutte le issue di Niko1221/Strata

Issue simili

Altre issue su C++

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.