Provider API: google/gemini-3.8-flash never gets prompt cache hits for structured-output requests (nested strict tool / response_format), while DeepSeek does

The issue identifies /provider/v1/chat/completions and provides a Python requests reproduction using Gemini with a nested strict tool schema; start by running that request twice and comparing prompt_tokens_details.cached_tokens. Then trace how the Provider API routes Gemini requests with tools or response_format, checking the related routing issue #1007. Done means the cause is identified and repeated structured-output requests report cache hits, or the upstream limitation is documented.

Summary

On the Provider API (/provider/v1/chat/completions), google/gemini-3.8-flash never reports a prompt cache hit for our production requests. These requests send a strict, forced tool call whose JSON schema contains nested objects, use stream: true, and repeat a long identical prefix a few seconds to a few minutes apart. Every call is billed at the full input rate (about 18k–24k input tokens per call, cached_tokens: 0, and the dashboard cost does not drop).

On the same account and route, deepseek/deepseek-v4.1-flash-fast with the same prompts and the same tool reports a ~61% cache share. Gemini plain requests (no tools, no response_format) do sometimes hit. So caching does work on this route, but not for structured-output requests in our tests.

Expected Behavior

Two identical requests (same messages, same tools) sent a few seconds apart should report prompt_tokens_details.cached_tokens > 0 on the second call and be billed at the cache-read rate, as they are for DeepSeek on this route.

Actual Behavior

Production-shaped requests (strict forced tool with nested objects, streaming) have never hit:

• 4 consecutive real game requests: ~18k–24k tokens, 1–4 minutes apart, identical system prompt and tool definition (we hash both), ~7–10k-token identical prefix. Result: 0 cached on all 4.
• Replay of 2 of those requests 60 s apart: 0, 0.
• Synthetic ~5.5k–9.5k-token fixed prompt with a nested-object tool, sent twice 5 s apart, in 5 separate pairs (streaming and non-streaming): 0 on every second call. This includes a tool whose only nesting is items: {type: object}.
• response_format: {type: json_schema} with a nested schema, and response_format: {type: json_object} (no tools), each sent twice 5 s apart: 0 on the second call.

Synthetic requests with no nested object did hit on the second call in several pairs (about 2k-token chunks):

Shape	Transport	cached_tokens on 2nd call
No tools, no response_format	non-stream	4,078 / 8,615
Strict tool, 1 string property	stream	4,068 / 8,727
Strict tool, 6 flat string properties	stream	4,035 / 8,155
Strict tool, flat + enum / length constraints	stream	4,024 / 6,103
No tools, 60 s apart, 3 calls	non-stream	0 → 4,077 → 4,077

These hits are not fully reliable, though. A later run of a flat 2-property tool non-streaming gave 0 on both calls (6,626 tokens). So we cannot say that schema shape alone decides it. It may also be routing (see #1007). What we can say for sure is that structured, nested-schema Gemini requests never hit in any of our runs, while DeepSeek with the same requests does.

prompt_cache_key gave no clear improvement in a separate 60-second-interval test with plain requests.

Steps to reproduce the issue

Production-shaped request, sent twice:

import time, requests
KEY = "..."  # Command Code API key
URL = "https://api.commandcode.ai/provider/v1/chat/completions"
rules = "\n".join(f"Clause {i}: the river mill {i % 7} keeps wheel {i * 3 % 13} turning and sack tally {i} recorded." for i in range(260))
S = {"type": "string"}
NESTED = {"type": "object", "additionalProperties": False, "required": ["lines"], "properties": {"lines": {"type": "array", "items": {
    "type": "object", "additionalProperties": False, "required": ["who", "say"], "properties": {"who": S, "say": S}}}}}

for i in range(2):
    body = {"model": "google/gemini-3.8-flash", "max_tokens": 16, "reasoning_effort": "low",
            "messages": [{"role": "system", "content": rules}, {"role": "user", "content": "Fill the tool briefly."}],
            "tools": [{"type": "function", "function": {"name": "emit", "strict": True, "parameters": NESTED}}],
            "tool_choice": {"type": "function", "function": {"name": "emit"}}, "parallel_tool_calls": False}
    u = requests.post(URL, headers={"Authorization": f"Bearer {KEY}"}, json=body, timeout=60).json()["usage"]
    print(i, u["prompt_tokens"], u["prompt_tokens_details"]["cached_tokens"])  # 2nd call: 0
    time.sleep(5)

For comparison: our production requests (same prompts and nested tool) sent to deepseek/deepseek-v4.1-flash-fast on this route reported 1,374,336 cached of 2,235,679 input tokens (61 of 69 calls with a hit).

Command Code Version

N/A — direct Provider API over HTTP (https://api.commandcode.ai/provider/v1), not the CLI

Operating System

Windows

Terminal/IDE

N/A (Python requests / Rust HTTP client)

Shell

PowerShell

Session file (optional)

No response

Fix prompt (optional)

No response

Additional context

• Times (UTC): real requests on 2026-10-29 between 18:21 and 18:30, synthetic tests between 18:43 and 19:00. Model google/gemini-3.8-flash, all calls returned HTTP 200.
• google/gemini-3.7-flash also showed 0 cached on repeated requests in our earlier tests.
• Possibly related: #1007 (unstable cache hits, routing suspected).
• Ask: please check whether implicit caching is applied to Gemini requests that use tools or response_format. If session-sticky routing is needed, please make it work for these requests too. If this is an upstream limitation, please document it, because the pricing page lists a Gemini cache-read rate.