Audit runs can leave .posthog-audit-checks.json in the user's project

Aperta Adatta ai principianti
#1,326 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
2/5
Tempo stimato
1-3 ore
Idoneità per principianti
75/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
typescript
Ambito
cli, tooling

Direzione di ricerca

Il file del registro di audit viene creato in src/lib/programs/audit/seed.ts e gestito in src/lib/programs/run-agent-legacy.ts. Esamina il blocco finally dove viene chiamato ledger.stop() e aggiungi un'eliminazione del file per programConfig.auditLedgerFile. Controlla anche src/agent/runner/harness/pi/security.ts per regolare l'ordine di registrazione per le rimozioni di file con ambito. Esegui il programma di audit localmente per verificare che il file venga ripulito e che non compaiano log fuorvianti 'bash denied'.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Problem

The wizard writes .posthog-audit-checks.json into the project root before the audit agent starts. No wizard code deletes it. The audit skills end with an instruction for the agent to run rm -f .posthog-audit-checks.json. The file is removed only if the agent runs that last step.

When the agent skips the step, the run still completes and the user keeps an untracked JSON file in their repo root, next to their own code. The next run does not break: the watcher ignores a leftover ledger and the seed replaces it. But the file is clutter, and a user can commit it by accident.

There is a second, smaller problem in the same path. On pi, a scoped rm of that file is logged as Denying bash command (not in allowlist) and captured as a bash denied event. Then the scoped-rm rule from #834 lets it run. The log and the telemetry say "denied". The command ran.

Evidence

Real-TUI sweep of the audit program on the wizard-workbench app audit/posthog-demo-3000. Harness pi (binding), sequence linear, model openai/gpt-5.6-sol.

  • Refactor stack B2 head d938a43b (#1307). The agent read step 6, uploaded the notebook, resolved write-report and upload-notebook, and published the handoff. It never ran the rm. After exit, .posthog-audit-checks.json (3929 bytes) was still in the project root.
  • Release A top 200961f8 (#1303). The agent ran rm -f .posthog-audit-checks.json and the file was removed. The wizard log shows the misleading pair:
    [pi] → bash {"command":"rm -f .posthog-audit-checks.json"}
    Denying bash command (not in allowlist): rm -f .posthog-audit-checks.json
    [pi] ← bash: {"content":[{"type":"text","text":"(no output)"}]}
    

The same prompt, harness and model gave both outcomes. The cleanup depends on the agent.

Code on main (d8486dc1):

  • src/lib/programs/audit/seed.ts:118-124: seedAuditLedger writes the ledger. src/lib/programs/audit/index.ts:26-29 and :58 call it before the agent starts. The wizard creates the file.
  • src/lib/programs/run-agent-legacy.ts:81-95: runProgramAgent starts the ledger watcher and stops it in finally. Nothing removes the file.
  • src/lib/programs/audit/ledger-watcher.ts:32: "A ledger an earlier run left behind stays ignored until this run writes." The code already expects leftovers.
  • src/lib/runners/run-wizard.ts:223: the task stream reads the checks from the session (getAuditChecks(session)), not from the file. Nothing reads the file after the run.
  • src/agent/runner/harness/pi/index.ts:741-749: pi deletes .posthog-events.json host-side after the run. There is no matching cleanup for the audit ledger.
  • src/agent/runner/harness/pi/runtime-notes.ts:53: pi still tells the agent that bash is only for install, build, typecheck, lint and format commands. #894 names this line as the reason agents skip rm. #1000 would update it.
  • src/agent/runner/harness/pi/security.ts:372-385: evaluateToolCall calls wizardCanUseTool first and applies isScopedFileRemoval after.
  • src/agent/agent-interface.ts:523-528: wizardCanUseTool logs Denying bash command and captures bash denied before it returns the deny that pi then overrides.

Skill side, context-mill main:

  • context/skills/audit/description.md:49 and context/skills/audit/references/6-report.md:284 tell the agent to delete the ledger.
  • The same step is in the six audit-* leaf skills, for example context/skills/audit-events/references/4-report.md:21, and in context/skills/events-audit/references/6-dashboard.md:270.
  • context-mill#306 (open) moves reports to publish_handoff and keeps the agent-driven delete.

The refactor stack has the same behavior. On d938a43b, src/programs/program-file-watchers.ts:16-58 owns the ledger for the run, but stop() only stops the watcher (src/programs/run-program.ts:706-708).

Prior work. #833 added a host-side cleanupArtifacts seam for .posthog-events.json. It was closed because it worked around pi blocking rm, and #834 fixed that instead. This case is different. rm works here, and the agent did not run it. The wizard also seeds this file, so the program is the file's owner. #898 and #901 fixed the misleading deny log and telemetry. Both were closed without merging, and main still has the old order. #612 (open) raises the same class of leak for /tmp scratch files that skills name themselves, and leans toward a skill-side fix because the wizard does not know those names. That concern does not apply here: the program declares the name in auditLedgerFile, so the wizard can delete it without learning skill filenames. #894 (open) would make the agent more likely to run the rm, but cleanup would still depend on the agent.

Suggested fix

  1. Delete the ledger in the wizard after the agent run settles, on every exit path. On main, remove path.join(session.installDir, programConfig.auditLedgerFile) with force: true after ledger.stop() in the finally of runProgramAgent, and in the registerCleanup callback so aborts clean up too. On the stack, do it in startProgramFileWatchers().stop() or next to it in the run-program.ts finally. This covers audit, events-audit and the family leaves, because they all set auditLedgerFile.
  2. In context-mill, remove the "delete .posthog-audit-checks.json" step from the audit skills, or mark it optional, once the wizard deletes the file. Then the skill and the wizard do not both own the file.
  3. In evaluateToolCall, decide the scoped-rm rule before the deny is logged and captured. Keep the disallowedTools check. Log an allow line for the command that runs, and do not capture bash denied for it. #898 has a small version of this change.
Lingua principale
TypeScript
Stelle
204
Fork
54
Merge medio
1g 5h
PR unite (30g)
51

Preparare l'ambiente

  • Nessun Dockerfile né file Docker Compose
  • Ha un modello di pull request
  • Nessuna guida per i contributori

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di PostHog/wizard

Tutte le issue di PostHog/wizard

Issue simili

Altre issue su TypeScript

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.