TestPromptGenerator: array-index ids silently skip new prompts when the prompts list is modified between runs

Đang mở
  • Có gì không ổn?
#66 0 bình luận 0 reaction 0 người được giao Xem trên GitHub

Chưa có ai nhận issue này.

Đánh giá

Độ khó
4/5
Thời gian dự kiến
3-5 ngày
Mức phù hợp với người mới
48/100
Loại issue
Lỗi
Độ rõ ràng
Khá rõ ràng
Mức độ hoạt động
Ít trao đổi
Công nghệ
typescript
Lĩnh vực
tooling

Hướng nghiên cứu

Start with src/prompt-generators/test.ts and compare its prompt ID behavior with the JSON generator's idField and the per-file generator. Review the README's test section and the loop-state behavior described in the reproduction, then add coverage for editing, reordering, and duplicate prompts. Done means resumed runs no longer silently skip or repeat prompts, or the documented limitation and recovery steps are explicit.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

bug S3

Observed behavior

TestPromptGenerator.generate assigns each prompt an id equal to its stringified position in the prompts array, and then asks loopState.isOutstanding(id) whether to yield it. The runtime loop persists the id of every completed prompt into the on-disk ${name}-loop-state.json so a resumed run can skip work that already finished.

When a user keeps the same loop name between runs (the normal case, since that is how resume works) and then edits the prompts array to add, remove, or reorder entries, the array index of every prompt that comes after the edit shifts. The completed ids on disk, however, are still the original '0', '1', .... The result is that:

  1. New prompts inserted at the front of the list are silently skipped because their (newly assigned) id was already completed under a previous run with different content.
  2. Existing prompts whose position shifted later in the list get yielded a second time, because their (newly assigned) id has not been completed yet.

Neither outcome produces a warning, a log line, or an error.

The source docstring and the README both describe this property as a feature: Prompt IDs are stable stringified array indices so loop state can skip completed prompts on resume. That is true while the array is unchanged, but the wording does not warn the caller that the contract assumes an immutable list. A reasonable user iterating on a test config and adding a new prompt to the top will silently lose work.

Expected behavior

Editing the prompts array should either:

  • Use ids that survive reordering (for example, a hash of the prompt text, or an explicit per-entry id field), so each prompt's completion state follows the prompt itself rather than the slot it currently occupies; or
  • Fail loudly when the prompts at completed ids no longer match what was previously processed (for example, by also persisting the prompt text alongside the id and detecting drift on resume); or
  • Document this footgun prominently next to the prompts field and in the test section of the README, so users know to delete the loop-state file when they edit the list.

The most defensive of these (content-derived ids, e.g. a short hash) matches what the JSON generator gets via idField and brings the test generator in line with per-file (where the id is the file path, which moves with the prompt).

Minimal reproduction

import { TestPromptGenerator } from 'loop-the-loop/prompt-generators/test';
import { LoopState } from 'loop-the-loop/util/loop-state';

// Simulate run 1: process 'A' and 'B'.
const stateRun1 = new LoopState('loop-state-ignore.json');
const gen1 = new TestPromptGenerator({ prompts: ['A', 'B'] });
for await (const p of gen1.generate(stateRun1)) {
  await stateRun1.end(p.id, { status: 'success', output: 'ok' });
}
// stateRun1 now has completed: ['0', '1'].

// User edits the config and adds a new prompt at the front: ['NEW', 'A', 'B'].
// On resume, the on-disk state still has completed: ['0', '1'].
const stateRun2 = new LoopState('loop-state-ignore.json', ['0', '1'], []);
const gen2 = new TestPromptGenerator({ prompts: ['NEW', 'A', 'B'] });
const yielded: Array<string> = [];
for await (const p of gen2.generate(stateRun2)) {
  yielded.push(`${p.id}:${p.prompt}`);
}
console.log(yielded);
// Actual:   ['2:B']         -- 'NEW' and 'A' silently skipped, 'B' re-run
// Expected: ['?:NEW']       -- only the genuinely new prompt yielded

Suggested fix

Derive the id from the prompt content rather than the array index. A short hash (for example, the first 12 hex chars of a SHA-256 of the prompt string) is enough to make ids survive reordering and additions without colliding in practice, and matches the JSON generator's optional idField story. If a duplicate prompt string is detected within a single config, fail fast with a clear error (similar to the proposal in #62) rather than silently skipping after the first one completes.

If a behavioural change is too risky, at minimum update the prompts docstring in src/prompt-generators/test.ts and the README's test section to spell out that editing the array between runs requires deleting the loop-state file.

Ngôn ngữ chính
TypeScript
Star
2
Fork
1
Không có pull request nào được merge trong 30 ngày

Chuẩn bị môi trường

Dự án này không cung cấp dev container, Dockerfile hay hướng dẫn đóng góp, nên bạn cần tự thiết lập môi trường: hãy bắt đầu từ README và xem hướng dẫn đóng góp lần đầu của chúng tôi để biết các bước chung.

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của joewalker/loop-the-loop

Tất cả issue của joewalker/loop-the-loop

Issue tương tự

Thêm issue về TypeScript

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.