TestPromptGenerator: array-index ids silently skip new prompts when the prompts list is modified between runs

Aperta
  • Qualcosa non va?
#66 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
48/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Tranquilla
Stack tecnologico
typescript
Ambito
tooling

Direzione di ricerca

Start with src/prompt-generators/test.ts and compare its prompt ID behavior with the JSON generator's idField and the per-file generator. Review the README's test section and the loop-state behavior described in the reproduction, then add coverage for editing, reordering, and duplicate prompts. Done means resumed runs no longer silently skip or repeat prompts, or the documented limitation and recovery steps are explicit.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

bug S3

Observed behavior

TestPromptGenerator.generate assigns each prompt an id equal to its stringified position in the prompts array, and then asks loopState.isOutstanding(id) whether to yield it. The runtime loop persists the id of every completed prompt into the on-disk ${name}-loop-state.json so a resumed run can skip work that already finished.

When a user keeps the same loop name between runs (the normal case, since that is how resume works) and then edits the prompts array to add, remove, or reorder entries, the array index of every prompt that comes after the edit shifts. The completed ids on disk, however, are still the original '0', '1', .... The result is that:

  1. New prompts inserted at the front of the list are silently skipped because their (newly assigned) id was already completed under a previous run with different content.
  2. Existing prompts whose position shifted later in the list get yielded a second time, because their (newly assigned) id has not been completed yet.

Neither outcome produces a warning, a log line, or an error.

The source docstring and the README both describe this property as a feature: Prompt IDs are stable stringified array indices so loop state can skip completed prompts on resume. That is true while the array is unchanged, but the wording does not warn the caller that the contract assumes an immutable list. A reasonable user iterating on a test config and adding a new prompt to the top will silently lose work.

Expected behavior

Editing the prompts array should either:

  • Use ids that survive reordering (for example, a hash of the prompt text, or an explicit per-entry id field), so each prompt's completion state follows the prompt itself rather than the slot it currently occupies; or
  • Fail loudly when the prompts at completed ids no longer match what was previously processed (for example, by also persisting the prompt text alongside the id and detecting drift on resume); or
  • Document this footgun prominently next to the prompts field and in the test section of the README, so users know to delete the loop-state file when they edit the list.

The most defensive of these (content-derived ids, e.g. a short hash) matches what the JSON generator gets via idField and brings the test generator in line with per-file (where the id is the file path, which moves with the prompt).

Minimal reproduction

import { TestPromptGenerator } from 'loop-the-loop/prompt-generators/test';
import { LoopState } from 'loop-the-loop/util/loop-state';

// Simulate run 1: process 'A' and 'B'.
const stateRun1 = new LoopState('loop-state-ignore.json');
const gen1 = new TestPromptGenerator({ prompts: ['A', 'B'] });
for await (const p of gen1.generate(stateRun1)) {
  await stateRun1.end(p.id, { status: 'success', output: 'ok' });
}
// stateRun1 now has completed: ['0', '1'].

// User edits the config and adds a new prompt at the front: ['NEW', 'A', 'B'].
// On resume, the on-disk state still has completed: ['0', '1'].
const stateRun2 = new LoopState('loop-state-ignore.json', ['0', '1'], []);
const gen2 = new TestPromptGenerator({ prompts: ['NEW', 'A', 'B'] });
const yielded: Array<string> = [];
for await (const p of gen2.generate(stateRun2)) {
  yielded.push(`${p.id}:${p.prompt}`);
}
console.log(yielded);
// Actual:   ['2:B']         -- 'NEW' and 'A' silently skipped, 'B' re-run
// Expected: ['?:NEW']       -- only the genuinely new prompt yielded

Suggested fix

Derive the id from the prompt content rather than the array index. A short hash (for example, the first 12 hex chars of a SHA-256 of the prompt string) is enough to make ids survive reordering and additions without colliding in practice, and matches the JSON generator's optional idField story. If a duplicate prompt string is detected within a single config, fail fast with a clear error (similar to the proposal in #62) rather than silently skipping after the first one completes.

If a behavioural change is too risky, at minimum update the prompts docstring in src/prompt-generators/test.ts and the README's test section to spell out that editing the array between runs requires deleting the loop-state file.

Lingua principale
TypeScript
Stelle
2
Fork
1
Nessuna PR unita negli ultimi 30g

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di joewalker/loop-the-loop

Tutte le issue di joewalker/loop-the-loop

Issue simili

Altre issue su TypeScript

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.