Cache the claude backend system prompt via Anthropic prompt caching

Closed Beginner friendly
  • Is something wrong?
#4,835 4 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

@Jvxoq is already working on this.

Since Oct 28, 2026.

  • #4855 by @Jvxoq — closed without merging

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
76/100
Issue type
Feature
Clarity
Clearly specified
Activity status
Active
Tech stack
python
Domain
ai, performance

Research direction

Start in graphify/llm.py at _call_claude() and review how the Anthropic SDK accepts system content blocks with cache_control. Apply caching to the extraction system prompt as described, then verify the call still uses the same prompt and message content. The issue defines done as enabling prompt caching for the native Claude backend without changing output.

Written by the indexing model from the issue text.

Description

Summary

extract --backend claude issues one messages.create call per file, and each call re-sends the fixed _extraction_system prompt at full input price. Marking the system block with Anthropic prompt caching (cache_control: ephemeral) bills that shared prefix at ~0.1× on reads after the first call. Output is unchanged — this is a billing/latency optimization only.

Where

graphify/llm.py, _call_claude() — system= is passed as a plain string, so the extraction system prompt is re-ingested on every per-file call.

Proposed change
diff
resp = client.messages.create(
model=model,
max_tokens=max_tokens,

  •    system=_extraction_system(deep=deep_mode),

  •    system=[{

  •        "type": "text",

  •        "text": _extraction_system(deep=deep_mode),

  •        "cache_control": {"type": "ephemeral"},

  •    }],
   messages=[{"role": "user", "content": _anthropic_content(user_message, images or [])}],
    )

The anthropic SDK already in use supports the content-block form of system with cache_control.

Why it helps

_extraction_system is identical across all per-file calls (only deep toggles). On a repo with N files the system prefix is paid N× at 1×; with caching it’s one write (1.25×) + (N−1) reads (0.1×) → ~90% off the system portion of input.

Caveats (for reviewers)
Only the system block is cacheable here; per-file user content isn’t reused.
Needs the system prompt ≥ the model’s min cacheable length (e.g. 512 tokens on recent Sonnet/Opus); below that it silently no-ops (cache_*_input_tokens = 0), no error.
With --max-concurrency > 1 the first wave of parallel calls misses (a cache entry is only available after the first response begins) — still a net win on multi-file runs.
5-min ephemeral TTL refreshes on each hit, so it stays warm through an active extraction.
Scoped to the native claude backend; the Bedrock path (system=[{"text": ...}]) uses a different caching mechanism and could be a follow-up.
Impact

Billing/latency only. No behavior change.

Dominant language
Python
Stars
124k
Forks
11.9k
Avg merge
2h 35m
Merged PRs (30d)
6

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from Graphify-Labs/graphify

All issues in Graphify-Labs/graphify

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.