<current_datetime> and random prompt_cache_key still block cross-session prompt caching (SDK 1.0.19 / runtime 1.0.95)

Aperta
  • Qualcosa non va?
#2,915 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
45/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
java, python
Ambito
backend-api-design

Direzione di ricerca

Start by reading the runtime request construction for current_datetime and prompt_cache_key, then trace the session options exposed by the SDK. Compare requests from separate sessions to confirm where the changing values enter the prompt. Done means adding a supported way to control the timestamp placement or precision, or to set a stable cache key, with coverage for requests that reuse a stable prompt prefix.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Follow-up to #1297, which was closed as not planned. That issue is still present in the current release and it blocks prompt caching across sessions for workloads that start a new session per request.

Environment

  • github-copilot-sdk 1.0.19 (Python), bundled runtime 1.0.95
  • Copilot billing, gpt-6-luna and gpt-6.1-sol
  • One new session per request, system_message={"mode": "replace", ...}, no tools, enable_session_store=False

What we measured

Five sequential requests with the same synthetic payload (11,454 input tokens each). Usage comes from assistant.usage.

Request Cached input Cache write
A (warm) 0 11,451
A repeated, identical 2,073 9,378
B, same prefix and different tail 2,073 9,378
A with a fixed working directory, twice 2,073 9,378
A with a fixed session_id, three times 2,073 9,378

The results were the same with both models. 2,073 tokens is exactly our system message. Nothing after it is ever read from cache.

What the runtime sends

We pointed a session at a local BYOK provider (type: "openai", wire_api: "responses") that only records requests. When we compared two sessions with the same prompt:

  1. The system message is byte-identical.
  2. The user message starts with <current_datetime>2026-11-11T00:04:49.196-03:00</current_datetime>, so the two requests differ from the first characters after the system message.
  3. prompt_cache_key is a different random value for each session, even when session_id is fixed.
  4. A blob attachment is placed after the datetime too, so no consumer content can come before it.

We found no option or environment variable that disables the datetime or controls prompt_cache_key.

Impact

This covers any consumer that sends a large stable prefix (shared rules, reference context) and opens a new session per request, such as a CI reviewer that splits one pull request into many requests. Those consumers pay for a cache write on every request and never get a cache read beyond the system message. The comment from November 3 on #1297 shows near-100% reuse once the timestamp is fixed, so the timestamp alone accounts for the loss.

Request

Any of the following would solve it for us:

  1. A session option to omit <current_datetime>, or to place it after the user content.
  2. Lower precision, for example the date or the hour, so requests close in time share the prefix.
  3. A way to set prompt_cache_key, or a value derived from the session configuration instead of a random one per session.
Lingua principale
Java
Stelle
10.7k
Fork
1.5k
Nessuna PR unita negli ultimi 30g

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di github/copilot-sdk

Tutte le issue di github/copilot-sdk

Issue simili

Altre issue su Java

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.