Hacktoberfest 2026：メンテナが10月に向けて印を付けた、オープンで初心者向けの issue。 Hacktoberfest の issue を見る

Let a one-shot request skip the prompt-cache work (~45 ms per call that is never reused)

クローズ
  • 何か問題がありますか？
#830 コメント 1 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

まだ誰も着手していません。

評価

難易度
4/5
見積もり時間
3〜5日
初心者へのやさしさ
52/100
issue の種類
機能追加
明瞭さ
おおむね明確
活発さ
活発
技術スタック
cpp, python
領域
ai-infra-agents, backend-api-design, performance

調査の方向性

Start by reading the prompt-cache flow in src/program/generate.cpp and how requests are handled in serve/server.py; the issue notes that serve/server.py does not currently read cache_prompt. Determine how a request-level opt-out should avoid the turn split and checkpoint without disabling cache reuse for other conversations. Done means one-shot requests can opt out, while interactive chats retain reuse; the CPU fallback and trace-counter questions are separate investigations.

索引モデルが issue の本文から書いたものです。

説明

A follow-up to part 2 of #519. Your explanation there of the fixed cost of the batched read (with about 40 % of the experts outside VRAM, a short batch still has to bring in nearly all the missing ones) matches what I see on 0.1.39: over 35 warm batched reads of 212 to 558 tokens the time fits about 245 ms + 0.33 ms per token. Following your suggestion, I moved the checkpoint to the end of the fixed instruction with --prompt-cache-root 64, which saves 20 to 40 ms per call. What is left on the cache side still costs about 45 ms per call.

Setup

  • RTX 5090 32 GB (no display), Ryzen 9 5950X (AVX2), 96 GB DDR4-3200, Windows 11
  • Strata 0.1.39 release, Swift IQ3_XXS, --expert-cache auto (14,864 slots, 24.2 GiB), --prefill auto:32768, --max-context 32768, --kv int8, --pcie-frac 0.20, --spec 4 --spec-min-p 0.70, --prompt-cache-root 256 for the numbers below (production now runs 64), STRATA_IQ_MT_MIN=1
  • The workload: a fixed system prompt of ~100 tokens, then a user message of 150 to 450 tokens of code, max_tokens 1, thinking off. Every call has a different user message, and the same server also takes interactive chats that do reuse their conversations.
  • The numbers below use the same shape built from public text: a 278-character system prompt (about 70 tokens) and excerpts of v0.1.32's src/program/generate.cpp, the same prompts in every run, 12 calls per size with the first 2 skipped, client on the same PC. I can share the script.

What the prompt cache costs on these calls

User message Prompt tokens read prompt_ms, cache on prompt_ms, --prompt-cache 0
~150 tokens 230 375 328
~300 tokens 379 434 388
~450 tokens 487 460 407

That is about 46 ms of prompt_ms and about 40 ms of wall time per call. With the cache on, the prompt is read in two parts, split at the last <|im_start|>, and a checkpoint is taken there. That checkpoint ends after the user message, which is different on every call, so nothing reuses it. With STRATA_TRACE=1, a 486-token prompt shows read 479 tokens (batched) in 403.5 ms, refilled 190 slots ... in 14.0 ms and read 6 tokens (windows) in 14.4 ms against prompt 486 tokens ... read in 460 ms. The trace leaves about 27 ms outside the timed reads, so the checkpoint itself costs at most that and the rest comes from the split.

--prompt-cache 0 removes this for the whole server, and the interactive chats lose their reuse with it. serve/server.py doesn't read cache_prompt, so a request can't ask for it.

Question

Could a request opt out of both the split at the last turn and the checkpoint when it has nothing to keep, for example by honouring cache_prompt: false, or automatically when a one-turn prompt's only reusable part is already a root? I'd expect it to save up to what --prompt-cache 0 saves on calls like these.

Separately: for a short batch where most of the missing experts are needed anyway, would running the non-resident ones on the CPU, as the decode windows do with misses, ever beat streaming them? If there's a trace counter that splits the batched read into transfer and compute time, I'd run it here.

I can share the traces and the script, and run a build with a change on this machine.

主要言語
C++
スター
11.6k
フォーク
1k
平均マージ
7時間 46分
マージ済み PR（30日）
30

環境構築

このプロジェクトには開発コンテナ、Dockerfile、コントリビューションガイドがありません。まず README を読み、一般的な手順ははじめてのコントリビューションガイドを参照してください。

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

Niko1221/Strata のほかの issue

Niko1221/Strata の issue をすべて見る

似ている issue

C++ の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。