Provider API: google/gemini-3.8-flash never gets prompt cache hits for structured-output requests (nested strict tool / response_format), while DeepSeek does

Ouverte
  • Quelque chose ne va pas ?
#1,089 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub

Les mainteneurs répondent en général sous 1 jour

Personne n'a encore pris cette issue.

Évaluation

Difficulté
4/5
Temps estimé
3-5 jours
Accessibilité débutants
42/100
Type d'issue
Bug
Clarté
Plutôt claire
Activité
Active
Stack technique
google-cloud, python
Domaine
api, backend

Piste de recherche

The issue identifies /provider/v1/chat/completions and provides a Python requests reproduction using Gemini with a nested strict tool schema; start by running that request twice and comparing prompt_tokens_details.cached_tokens. Then trace how the Provider API routes Gemini requests with tools or response_format, checking the related routing issue #1007. Done means the cause is identified and repeated structured-output requests report cache hits, or the upstream limitation is documented.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Description

Summary

On the Provider API (/provider/v1/chat/completions), google/gemini-3.8-flash never reports a prompt cache hit for our production requests. These requests send a strict, forced tool call whose JSON schema contains nested objects, use stream: true, and repeat a long identical prefix a few seconds to a few minutes apart. Every call is billed at the full input rate (about 18k–24k input tokens per call, cached_tokens: 0, and the dashboard cost does not drop).

On the same account and route, deepseek/deepseek-v4.1-flash-fast with the same prompts and the same tool reports a ~61% cache share. Gemini plain requests (no tools, no response_format) do sometimes hit. So caching does work on this route, but not for structured-output requests in our tests.

Expected Behavior

Two identical requests (same messages, same tools) sent a few seconds apart should report prompt_tokens_details.cached_tokens > 0 on the second call and be billed at the cache-read rate, as they are for DeepSeek on this route.

Actual Behavior

Production-shaped requests (strict forced tool with nested objects, streaming) have never hit:

  • 4 consecutive real game requests: ~18k–24k tokens, 1–4 minutes apart, identical system prompt and tool definition (we hash both), ~7–10k-token identical prefix. Result: 0 cached on all 4.
  • Replay of 2 of those requests 60 s apart: 0, 0.
  • Synthetic ~5.5k–9.5k-token fixed prompt with a nested-object tool, sent twice 5 s apart, in 5 separate pairs (streaming and non-streaming): 0 on every second call. This includes a tool whose only nesting is items: {type: object}.
  • response_format: {type: json_schema} with a nested schema, and response_format: {type: json_object} (no tools), each sent twice 5 s apart: 0 on the second call.

Synthetic requests with no nested object did hit on the second call in several pairs (about 2k-token chunks):

Shape Transport cached_tokens on 2nd call
No tools, no response_format non-stream 4,078 / 8,615
Strict tool, 1 string property stream 4,068 / 8,727
Strict tool, 6 flat string properties stream 4,035 / 8,155
Strict tool, flat + enum / length constraints stream 4,024 / 6,103
No tools, 60 s apart, 3 calls non-stream 0 → 4,077 → 4,077

These hits are not fully reliable, though. A later run of a flat 2-property tool non-streaming gave 0 on both calls (6,626 tokens). So we cannot say that schema shape alone decides it. It may also be routing (see #1007). What we can say for sure is that structured, nested-schema Gemini requests never hit in any of our runs, while DeepSeek with the same requests does.

prompt_cache_key gave no clear improvement in a separate 60-second-interval test with plain requests.

Steps to reproduce the issue

Production-shaped request, sent twice:

import time, requests
KEY = "..."  # Command Code API key
URL = "https://api.commandcode.ai/provider/v1/chat/completions"
rules = "\n".join(f"Clause {i}: the river mill {i % 7} keeps wheel {i * 3 % 13} turning and sack tally {i} recorded." for i in range(260))
S = {"type": "string"}
NESTED = {"type": "object", "additionalProperties": False, "required": ["lines"], "properties": {"lines": {"type": "array", "items": {
    "type": "object", "additionalProperties": False, "required": ["who", "say"], "properties": {"who": S, "say": S}}}}}

for i in range(2):
    body = {"model": "google/gemini-3.8-flash", "max_tokens": 16, "reasoning_effort": "low",
            "messages": [{"role": "system", "content": rules}, {"role": "user", "content": "Fill the tool briefly."}],
            "tools": [{"type": "function", "function": {"name": "emit", "strict": True, "parameters": NESTED}}],
            "tool_choice": {"type": "function", "function": {"name": "emit"}}, "parallel_tool_calls": False}
    u = requests.post(URL, headers={"Authorization": f"Bearer {KEY}"}, json=body, timeout=60).json()["usage"]
    print(i, u["prompt_tokens"], u["prompt_tokens_details"]["cached_tokens"])  # 2nd call: 0
    time.sleep(5)

For comparison: our production requests (same prompts and nested tool) sent to deepseek/deepseek-v4.1-flash-fast on this route reported 1,374,336 cached of 2,235,679 input tokens (61 of 69 calls with a hit).

Command Code Version

N/A — direct Provider API over HTTP (https://api.commandcode.ai/provider/v1), not the CLI

Operating System

Windows

Terminal/IDE

N/A (Python requests / Rust HTTP client)

Shell

PowerShell

Session file (optional)

No response

Fix prompt (optional)

No response

Additional context
  • Times (UTC): real requests on 2026-10-29 between 18:21 and 18:30, synthetic tests between 18:43 and 19:00. Model google/gemini-3.8-flash, all calls returned HTTP 200.
  • google/gemini-3.7-flash also showed 0 cached on repeated requests in our earlier tests.
  • Possibly related: #1007 (unstable cache hits, routing suspected).
  • Ask: please check whether implicit caching is applied to Gemini requests that use tools or response_format. If session-sticky routing is needed, please make it work for these requests too. If this is an upstream limitation, please document it, because the pricing page lists a Gemini cache-read rate.
Langage dominant
Aucune donnée de langage
Étoiles
4.2k
Forks
377
Merge moyen
1 j 3 h
PR mergées (30 j)
6

Préparer son environnement

Ce projet ne fournit ni conteneur de développement, ni Dockerfile, ni guide de contribution : l'installation est à votre charge. Commencez par son README, et consultez notre guide de la première contribution pour les étapes générales.

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Autres issues de CommandCodeAI/command-code

Toutes les issues de CommandCodeAI/command-code

Issues similaires

Plus d'issues Backend & API Design

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.