Outro names a report file the agent was told not to write

Open
#1,332 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
65/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
typescript
Domain
cli, documentation

Research direction

Start by reading the conflicting instructions in src/agent/tools/handoff.ts and src/agent/agent-prompt.ts. Then examine the outro logic in src/agent/runner/sequence/linear.ts and src/ui/tui/screens/OutroScreen.tsx to see where the report file name is used. The fix involves aligning the instructions and conditionally displaying the file path only if the file exists. Run a test program to verify the behavior.

Written by the indexing model from the issue text.

Description

Problem

Main now has part of the "report goes through publish_handoff" change, and the outros point at files that do not exist.

#1141 (merged 2026-08-27) added this clause to the publish_handoff description: "do not write the report to a file yourself". It is at src/agent/tools/handoff.ts:30-33 on main (d8486dc1), and the pi tool uses the same text. The rest of the code still asks for the file or says it exists:

  • skillPrompt still says "write a brief markdown report to ./${reportFile}" (src/agent/agent-prompt.ts:54). Every program with a skillId gets this line.
  • The context-mill skills still say to write it. context/skills/data-warehouse-source/description.md:99 says "write the report file requested by the wizard". context/skills/audit/references/6-report.md:7 and :52 say to Write posthog-audit-report.md to disk. context-mill#306 moves these skills to publish_handoff. It is still a draft.
  • The outros show the file name and never check the file exists. src/agent/runner/sequence/linear.ts:294-300 passes run.reportFile through. src/ui/tui/screens/OutroScreen.tsx:87-93 prints "Check ./… for details", and src/ui/tui/exit-line.ts:91-92 adds the same line on exit. src/ui/tui/screens/audit/AuditOutroScreen.tsx:64-76 prints "Report saved to:". The audit success message at src/lib/programs/audit/index.ts:48, the ledger row label at src/lib/programs/audit/seed.ts:99-103 and the "Write report" slide at src/ui/tui/screens/audit/slides/writeReport.tsx:25 also name the file.
  • On main, publish_handoff writes a file only when the host sets POSTHOG_HANDOFF_OUTPUT_PATH. The TUI does not set it, so nothing is written to the project.
  • The wizard-workbench warehouse grader also expects the file. Its "report written" check passes only when result.reportFile.exists is true (PostHog/wizard-workbench services/wizard-ci/warehouse-checks.ts:813-820), so a run that follows the tool description fails it.

Evidence

Each run follows one instruction or the other. We saw this in a real-TUI sweep of all nine programs on the refactor stack head d938a43b (#1307), which has the same code for this path as main:

  • warehouse (revenue/stripe/stripe-saas-demo): the agent called only publish_handoff and marked "Write setup report" completed. The outro says "Check ./posthog-warehouse-report.md for details". No file was written.
  • audit (audit/posthog-demo-3000): the agent resolved write-report as pass: "Final markdown report delivered through the wizard handoff." The outro says "Report saved to: /posthog-audit-report.md". No file was written. The Release A top (200961f8) did the same.
  • metrics (express-todo, orchestrator flow): the report went only to a notebook. The orchestrator outro does not name a file, so it shows nothing false.
  • ai-observability wrote its file and also called publish_handoff. On 200961f8, warehouse wrote its file too. The result depends on which instruction the model follows.

The stack carries the same lines at other positions (src/agent/agent-prompt.ts:54, src/agent/tools/handoff.ts:33, src/agent/runner/sequence/linear.ts:314, src/ui/tui/screens/OutroScreen.tsx:107, src/ui/tui/screens/audit/AuditOutroScreen.tsx:67, src/programs/resolve-run-definition.ts:56), so this is not a stack regression.

Prior work

#1047 (open draft, last updated 2026-07-31, conflicting) makes publish_handoff the only report channel. It changes skillPrompt, stops programs naming a .md file, and shows the file in the outro only when one was written. context-mill#306 is its skill-side pair. Both are stale, and #1047 predates #1141, so it does not describe the contradiction main has now. Its base is ce8b70d3, and the files have moved since: src/lib/agent is now src/agent, and src/lib/wizard-tools is now src/agent/tools.

#811 (open draft, stale since 2026-07-06) gates the linear-runner outro on the file existing. It is on old paths and does not cover the audit success message or screen.

Suggested fix

  1. Now: pick one instruction. Either drop "do not write the report to a file yourself" from PUBLISH_HANDOFF_DESCRIPTION until the rest moves, or change skillPrompt to order a publish_handoff call and stop naming the file.
  2. Until one channel remains, show a report path only when that file exists in installDir. #811 did this for the linear runner. The same check needs to cover the audit outro screen, the audit success message and the exit line.
  3. Longer term: land #1047 and context-mill#306 together, or close them and do the same change on the current paths. Update the workbench "report written" check to accept the published handoff when the program no longer names a file.
Dominant language
TypeScript
Stars
204
Forks
54
Avg merge
1d 5h
Merged PRs (30d)
51

Getting set up

  • No Dockerfile or Docker Compose file
  • Has a pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from PostHog/wizard

All issues in PostHog/wizard

Similar issues

More TypeScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.