TestPromptGenerator: array-index ids silently skip new prompts when the prompts list is modified between runs

Abierto
  • ¿Algo no está bien?
#66 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
48/100
Tipo de issue
Error
Claridad
Bastante claro
Estado de actividad
Tranquilo
Stack tecnológico
typescript
Área
tooling

Línea de trabajo

Start with src/prompt-generators/test.ts and compare its prompt ID behavior with the JSON generator's idField and the per-file generator. Review the README's test section and the loop-state behavior described in the reproduction, then add coverage for editing, reordering, and duplicate prompts. Done means resumed runs no longer silently skip or repeat prompts, or the documented limitation and recovery steps are explicit.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

bug S3

Observed behavior

TestPromptGenerator.generate assigns each prompt an id equal to its stringified position in the prompts array, and then asks loopState.isOutstanding(id) whether to yield it. The runtime loop persists the id of every completed prompt into the on-disk ${name}-loop-state.json so a resumed run can skip work that already finished.

When a user keeps the same loop name between runs (the normal case, since that is how resume works) and then edits the prompts array to add, remove, or reorder entries, the array index of every prompt that comes after the edit shifts. The completed ids on disk, however, are still the original '0', '1', .... The result is that:

  1. New prompts inserted at the front of the list are silently skipped because their (newly assigned) id was already completed under a previous run with different content.
  2. Existing prompts whose position shifted later in the list get yielded a second time, because their (newly assigned) id has not been completed yet.

Neither outcome produces a warning, a log line, or an error.

The source docstring and the README both describe this property as a feature: Prompt IDs are stable stringified array indices so loop state can skip completed prompts on resume. That is true while the array is unchanged, but the wording does not warn the caller that the contract assumes an immutable list. A reasonable user iterating on a test config and adding a new prompt to the top will silently lose work.

Expected behavior

Editing the prompts array should either:

  • Use ids that survive reordering (for example, a hash of the prompt text, or an explicit per-entry id field), so each prompt's completion state follows the prompt itself rather than the slot it currently occupies; or
  • Fail loudly when the prompts at completed ids no longer match what was previously processed (for example, by also persisting the prompt text alongside the id and detecting drift on resume); or
  • Document this footgun prominently next to the prompts field and in the test section of the README, so users know to delete the loop-state file when they edit the list.

The most defensive of these (content-derived ids, e.g. a short hash) matches what the JSON generator gets via idField and brings the test generator in line with per-file (where the id is the file path, which moves with the prompt).

Minimal reproduction

import { TestPromptGenerator } from 'loop-the-loop/prompt-generators/test';
import { LoopState } from 'loop-the-loop/util/loop-state';

// Simulate run 1: process 'A' and 'B'.
const stateRun1 = new LoopState('loop-state-ignore.json');
const gen1 = new TestPromptGenerator({ prompts: ['A', 'B'] });
for await (const p of gen1.generate(stateRun1)) {
  await stateRun1.end(p.id, { status: 'success', output: 'ok' });
}
// stateRun1 now has completed: ['0', '1'].

// User edits the config and adds a new prompt at the front: ['NEW', 'A', 'B'].
// On resume, the on-disk state still has completed: ['0', '1'].
const stateRun2 = new LoopState('loop-state-ignore.json', ['0', '1'], []);
const gen2 = new TestPromptGenerator({ prompts: ['NEW', 'A', 'B'] });
const yielded: Array<string> = [];
for await (const p of gen2.generate(stateRun2)) {
  yielded.push(`${p.id}:${p.prompt}`);
}
console.log(yielded);
// Actual:   ['2:B']         -- 'NEW' and 'A' silently skipped, 'B' re-run
// Expected: ['?:NEW']       -- only the genuinely new prompt yielded

Suggested fix

Derive the id from the prompt content rather than the array index. A short hash (for example, the first 12 hex chars of a SHA-256 of the prompt string) is enough to make ids survive reordering and additions without colliding in practice, and matches the JSON generator's optional idField story. If a duplicate prompt string is detected within a single config, fail fast with a clear error (similar to the proposal in #62) rather than silently skipping after the first one completes.

If a behavioural change is too risky, at minimum update the prompts docstring in src/prompt-generators/test.ts and the README's test section to spell out that editing the array between runs requires deleting the loop-state file.

Lenguaje dominante
TypeScript
Estrellas
2
Forks
1
Sin PR fusionados en 30 d

Preparar el entorno

Este proyecto no incluye contenedor de desarrollo, Dockerfile ni guía de contribución, así que la configuración corre por tu cuenta: empieza por su README y consulta nuestra guía para la primera contribución para los pasos generales.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de joewalker/loop-the-loop

Todos los issues de joewalker/loop-the-loop

Issues similares

Más issues de TypeScript

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.