Provider API: google/gemini-3.8-flash never gets prompt cache hits for structured-output requests (nested strict tool / response_format), while DeepSeek does

Abierto
  • ¿Algo no está bien?
#1,089 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
42/100
Tipo de issue
Error
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
google-cloud, python
Área
api, backend

Línea de trabajo

The issue identifies /provider/v1/chat/completions and provides a Python requests reproduction using Gemini with a nested strict tool schema; start by running that request twice and comparing prompt_tokens_details.cached_tokens. Then trace how the Provider API routes Gemini requests with tools or response_format, checking the related routing issue #1007. Done means the cause is identified and repeated structured-output requests report cache hits, or the upstream limitation is documented.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Summary

On the Provider API (/provider/v1/chat/completions), google/gemini-3.8-flash never reports a prompt cache hit for our production requests. These requests send a strict, forced tool call whose JSON schema contains nested objects, use stream: true, and repeat a long identical prefix a few seconds to a few minutes apart. Every call is billed at the full input rate (about 18k–24k input tokens per call, cached_tokens: 0, and the dashboard cost does not drop).

On the same account and route, deepseek/deepseek-v4.1-flash-fast with the same prompts and the same tool reports a ~61% cache share. Gemini plain requests (no tools, no response_format) do sometimes hit. So caching does work on this route, but not for structured-output requests in our tests.

Expected Behavior

Two identical requests (same messages, same tools) sent a few seconds apart should report prompt_tokens_details.cached_tokens > 0 on the second call and be billed at the cache-read rate, as they are for DeepSeek on this route.

Actual Behavior

Production-shaped requests (strict forced tool with nested objects, streaming) have never hit:

  • 4 consecutive real game requests: ~18k–24k tokens, 1–4 minutes apart, identical system prompt and tool definition (we hash both), ~7–10k-token identical prefix. Result: 0 cached on all 4.
  • Replay of 2 of those requests 60 s apart: 0, 0.
  • Synthetic ~5.5k–9.5k-token fixed prompt with a nested-object tool, sent twice 5 s apart, in 5 separate pairs (streaming and non-streaming): 0 on every second call. This includes a tool whose only nesting is items: {type: object}.
  • response_format: {type: json_schema} with a nested schema, and response_format: {type: json_object} (no tools), each sent twice 5 s apart: 0 on the second call.

Synthetic requests with no nested object did hit on the second call in several pairs (about 2k-token chunks):

Shape Transport cached_tokens on 2nd call
No tools, no response_format non-stream 4,078 / 8,615
Strict tool, 1 string property stream 4,068 / 8,727
Strict tool, 6 flat string properties stream 4,035 / 8,155
Strict tool, flat + enum / length constraints stream 4,024 / 6,103
No tools, 60 s apart, 3 calls non-stream 0 → 4,077 → 4,077

These hits are not fully reliable, though. A later run of a flat 2-property tool non-streaming gave 0 on both calls (6,626 tokens). So we cannot say that schema shape alone decides it. It may also be routing (see #1007). What we can say for sure is that structured, nested-schema Gemini requests never hit in any of our runs, while DeepSeek with the same requests does.

prompt_cache_key gave no clear improvement in a separate 60-second-interval test with plain requests.

Steps to reproduce the issue

Production-shaped request, sent twice:

import time, requests
KEY = "..."  # Command Code API key
URL = "https://api.commandcode.ai/provider/v1/chat/completions"
rules = "\n".join(f"Clause {i}: the river mill {i % 7} keeps wheel {i * 3 % 13} turning and sack tally {i} recorded." for i in range(260))
S = {"type": "string"}
NESTED = {"type": "object", "additionalProperties": False, "required": ["lines"], "properties": {"lines": {"type": "array", "items": {
    "type": "object", "additionalProperties": False, "required": ["who", "say"], "properties": {"who": S, "say": S}}}}}

for i in range(2):
    body = {"model": "google/gemini-3.8-flash", "max_tokens": 16, "reasoning_effort": "low",
            "messages": [{"role": "system", "content": rules}, {"role": "user", "content": "Fill the tool briefly."}],
            "tools": [{"type": "function", "function": {"name": "emit", "strict": True, "parameters": NESTED}}],
            "tool_choice": {"type": "function", "function": {"name": "emit"}}, "parallel_tool_calls": False}
    u = requests.post(URL, headers={"Authorization": f"Bearer {KEY}"}, json=body, timeout=60).json()["usage"]
    print(i, u["prompt_tokens"], u["prompt_tokens_details"]["cached_tokens"])  # 2nd call: 0
    time.sleep(5)

For comparison: our production requests (same prompts and nested tool) sent to deepseek/deepseek-v4.1-flash-fast on this route reported 1,374,336 cached of 2,235,679 input tokens (61 of 69 calls with a hit).

Command Code Version

N/A — direct Provider API over HTTP (https://api.commandcode.ai/provider/v1), not the CLI

Operating System

Windows

Terminal/IDE

N/A (Python requests / Rust HTTP client)

Shell

PowerShell

Session file (optional)

No response

Fix prompt (optional)

No response

Additional context
  • Times (UTC): real requests on 2026-10-29 between 18:21 and 18:30, synthetic tests between 18:43 and 19:00. Model google/gemini-3.8-flash, all calls returned HTTP 200.
  • google/gemini-3.7-flash also showed 0 cached on repeated requests in our earlier tests.
  • Possibly related: #1007 (unstable cache hits, routing suspected).
  • Ask: please check whether implicit caching is applied to Gemini requests that use tools or response_format. If session-sticky routing is needed, please make it work for these requests too. If this is an upstream limitation, please document it, because the pricing page lists a Gemini cache-read rate.
Lenguaje dominante
Sin datos de lenguaje
Estrellas
4.2k
Forks
377
Merge medio
1 d 3 h
PR fusionados (30 d)
6

Preparar el entorno

Este proyecto no incluye contenedor de desarrollo, Dockerfile ni guía de contribución, así que la configuración corre por tu cuenta: empieza por su README y consulta nuestra guía para la primera contribución para los pasos generales.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de CommandCodeAI/command-code

Todos los issues de CommandCodeAI/command-code

Issues similares

Más issues de Backend & API Design

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.