Model-call cache barely replays on a re-run of the same head, and identical input yields disjoint findings

Abierto
#30 1 comentario 0 reacciones 0 asignados Ver en GitHub

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
48/100
Tipo de issue
Error
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
typescript

Línea de trabajo

No files or tests are named. Start by tracing the model-call cache key and the run.json call and token totals, then reproduce the cold and warm runs described in the report. Done means identifying why identical packets miss the cache and making cache hit and miss counts available to callers.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Problem

Re-reviewing the same head should be close to free: the diff is identical, the packets are identical, and only the reviewer's prior findings differ. In practice a warm run costs almost as much as a cold one.

A/B on 0.5.6, same repository, same head commit, same provider and model, both invocations with --cache, run back to back:

run model calls total tokens
cold 96 2.46M
warm 80 2.17M

That is 17% fewer calls and 12% fewer tokens - within the run-to-run noise of this pipeline rather than a replay.

The second signal is more interesting than the cost: the two runs produced disjoint findings on identical input. Same head, same model, same configuration, no overlap in what came back.

Questions
  1. What exactly is the model-call cache keyed on? If the packet prompt embeds anything that varies per run - ordering of retrieved context, timestamps, a run id, a shuffled symbol list - the key changes on every invocation and the cache can never hit on the work that matters.
  2. Are cache hits and misses counted anywhere a caller can read them? run.json reports call and token totals, but nothing I could find states how many calls were served from the cache. Without that counter, a user cannot tell a cold pipeline from a broken cache - which is exactly the position this report is written from.
Why it matters

Review rounds repeat. A change is pushed, the reviewer runs again, and most packets are byte-identical to the previous round. If those replay, the marginal cost of an extra round approaches zero and re-reviewing becomes routine. If they do not, every round is charged at full price and teams ration the tool.

The disjoint-findings result also makes caching a correctness question, not only a cost one: if identical input can produce non-overlapping output, then a cache hit and a cache miss are not interchangeable, and it matters which one a user got.

Offer

I can re-run the A/B on 0.5.7 with any additional telemetry you want emitted, and report per-packet hit/miss if the counters are exposed.

Lenguaje dominante
TypeScript
Estrellas
8
Forks
1
Merge medio
5 min
PR fusionados (30 d)
2

Preparar el entorno

Este proyecto no incluye contenedor de desarrollo, Dockerfile ni guía de contribución, así que la configuración corre por tu cuenta: empieza por su README y consulta nuestra guía para la primera contribución para los pasos generales.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de 0xPolygon/codegenie

Todos los issues de 0xPolygon/codegenie

Issues similares

Más issues de TypeScript

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.