Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

PerFilePromptGenerator: prompt template includes are re-read from disk once per file

Aperta Adatta ai principianti
  • Qualcosa non va?
#65 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
2/5
Tempo stimato
1-3 ore
Idoneità per principianti
75/100
Tipo di issue
Refactoring
Chiarezza
Specificata chiaramente
Stato di attività
Tranquilla
Stack tecnologico
typescript
Ambito
performance

Direzione di ricerca

Inizia in src/prompt-generators/per-file.ts e src/util/expand-prompt.ts, quindi esegui la riproduzione di dry-run fornita oppure una riproduzione del generatore a livello di unità contando le chiamate a readFile. Sposta l’espansione degli include al di fuori del ciclo per file, mantenendo la sostituzione dei file per file; il lavoro è completato quando i prompt generati non cambiano e il file include viene letto una volta per ogni chiamata a generate().

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

bug S3

Observed behavior

PerFilePromptGenerator.generate() calls expandPrompt(template, basePath, { file }) inside the per-file loop. expandPrompt runs expandIncludes first (which recursively readFiles every {{include:path}} target) and only then substitutes the variables.

Source: src/prompt-generators/per-file.ts:67-78

async *generate(loopState: LoopState): AsyncIterable<Prompt> {
  const { filePattern, excludePatterns } = this.#task;
  const allFiles = await resolveFiles(filePattern, excludePatterns);

  for (const file of allFiles) {
    if (loopState.isOutstanding(file)) {
      const template = this.#task.promptTemplate;
      const prompt = await expandPrompt(template, this.#basePath, { file });
      yield { id: file, prompt };
    }
  }
}

And src/util/expand-prompt.ts:17-29:

export async function expandPrompt(
  template: string,
  basePath: string,
  variables: Readonly<Record<string, string>>,
): Promise<string> {
  let result = await expandIncludes(template, basePath);
  for (const [key, value] of Object.entries(variables)) {
    result = result.replaceAll(`{{${key}}}`, value);
  }
  return result;
}

The only per-file variable is file. Includes do not, and cannot, depend on it: expandIncludes runs strictly before variable substitution. So the resolved-includes form of the template is identical across every iteration of the loop, yet it is recomputed (and the include files re-read) for every file.

For the project's own src/examples/quality-check/quality-check.json config, which uses {{include:prompt.md}} against filePattern: src/**/*.ts, every prompt re-reads prompt.md even though its contents never change between iterations. Larger include trees and bigger file sets amplify the wasted I/O linearly. With recursive includes the multiplier compounds, since expandIncludes walks the include graph again from scratch on every call.

Expected behavior

Expand {{include:...}} macros once per generate() call, cache the result, and only run the {{file}} substitution per iteration. The include graph is fixed for the lifetime of a run, so this is a safe optimization with the same observable output.

Minimal reproduction

mkdir -p /tmp/llt-include-repro/src
for i in $(seq 1 200); do touch /tmp/llt-include-repro/src/f$i.ts; done

# A non-trivial include file
cat > /tmp/llt-include-repro/prompt.md <<'MD'
Review {{file}} for issues.
MD

cat > /tmp/llt-include-repro/config.json <<'JSON'
{
  "name": "include-perf",
  "agent": "claude-sdk",
  "promptGenerator": [
    "per-file",
    {
      "filePattern": "src/**/*.ts",
      "promptTemplate": "{{include:prompt.md}}"
    }
  ]
}
JSON

# Use --dry-run so we exercise only prompt generation, then strace/instrument the
# readFile calls to prompt.md. With 200 files there are 200 readFile calls for
# prompt.md; one is sufficient.
cd /tmp/llt-include-repro
pnpx loop-the-loop --dry-run config.json

Equivalently, a unit-level repro: wrap node:fs/promises.readFile with a counter, run PerFilePromptGenerator.generate() over N files where the template is {{include:prompt.md}}, and observe N reads of prompt.md instead of 1.

Suggested fix

Move the include expansion out of the loop:

async *generate(loopState: LoopState): AsyncIterable<Prompt> {
  const { filePattern, excludePatterns, promptTemplate } = this.#task;
  const allFiles = await resolveFiles(filePattern, excludePatterns);
  const expandedTemplate = await expandIncludes(promptTemplate, this.#basePath);

  for (const file of allFiles) {
    if (loopState.isOutstanding(file)) {
      const prompt = expandedTemplate.replaceAll('{{file}}', file);
      yield { id: file, prompt };
    }
  }
}

This preserves the existing behavior for templates with no includes (one cheap expandIncludes call up front instead of N), and avoids re-reading any included files for every iteration. The same optimization applies to other generators that call expandPrompt in a loop (json, bugzilla, github, gitlab) and is worth considering uniformly, but per-file is the one this issue is about. A unified alternative would be to add an expandPromptWithCachedIncludes (or an opt-in precomputeIncludes) helper in src/util/expand-prompt.ts and use it from all looping generators.

Note this is purely a performance/maintainability fix: callers see the same expanded text either way, so no behavioral migration is required.

Lingua principale
TypeScript
Stelle
2
Fork
1
Nessuna PR unita negli ultimi 30g

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di joewalker/loop-the-loop

Tutte le issue di joewalker/loop-the-loop

Issue simili

Altre issue su TypeScript

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.