Track multi-route AI observability coverage and blocked evaluator recheck
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 35/100
- Issue type
- Bug
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- typescript
- Domain
- observability, testing
Research direction
Start in services/pr-evaluator/evaluator.ts, focusing on how successive SDK success messages are handled and how the evaluator report is retained. Run pnpm test:evaluator, then rerun the updated rubric against the saved before/after diffs. Done means the complete report is preserved and the six route groups are checked for complete $ai_output_choices payloads, including role, plus the SSE variants and coverage report.
Written by the indexing model from the issue text.
Description
Status
Parked for now. This tracks the multi-route AI observability reproduction and the blocked before/after evaluation. No further runs are planned until work resumes.
Reproduction and findings
The Workbench fixture PR exercises six inference route groups: OpenAI-compatible chat, Responses, Ollama chat and generate, direct assistant tool calls, and WebSocket generation. It includes JSON, SSE, NDJSON, provider failures, and cancellation.
- Before guidance changes, the Wizard run took 236.5 seconds and incorrectly captured a successful WebSocket generation as an error without output or tokens.
- After applying Context Mill #406 and manually supplying the Markdown from docs #20409, the Wizard run took 352.5 seconds. The WebSocket classification improved, but five successful
$ai_output_choicesentries across four route groups still lackrole. The promised inference coverage ledger/report was not written, and the SSE completion parser remains brittle to valid JSON whitespace. - Both runs added capture hooks for all six route groups and passed the fixture's five response-contract tests. Those checks do not establish that the emitted AI observability payloads are complete.
The earlier gateway evaluator scored both runs 5/5 despite the remaining payload/report misses. The Workbench PR adds specific rubric criteria and scoring caps; pnpm test:evaluator passed 57/57. The live recheck with that rubric did not yield a usable after score.
Evaluation blocker
In the live recheck, the evaluator logged a completed report result (10,784 output tokens), then another success result (28 output tokens). Its loop assigns resultText = message.result on each success, so the later result appears to replace the full report. It then retried and failed with No result received from agent (including retry). A model mismatch is possible but unconfirmed; the observed result replacement should be investigated first. This is in the Workbench evaluator (services/pr-evaluator/evaluator.ts), not established as the same failure as Wizard's long-running gateway connection issue #581.
When resumed
- Preserve the complete evaluator report when subsequent SDK result messages arrive, and rerun the updated rubric against the saved before/after diffs.
- Verify each successful generation has the required
$ai_output_choicesshape, includingrole, across all six routes; verify SSE variants and the promised coverage ledger/report. - Decide whether any remaining misses need Context Mill guidance, docs, or Wizard behavior changes, using the Workbench fixture as the regression case.
- Dominant language
- TypeScript
- Stars
- 204
- Forks
- 54
- Avg merge
- 1d 5h
- Merged PRs (30d)
- 51
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- No contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from PostHog/wizard
-
Difficulty 2/5 1-3 hours Newbie friendliness 66/100
PostHog/wizard#1460 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
Maintainers usually reply within 1 day
-
Self-driving setup: checkbox labels and Confirm button overlap wrapped description textMay be free again A pull request for this issue was closed without being merged. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
PostHog/wizard#986 · 4 comments ·
Maintainers usually reply within 1 day
-
Intro shows "(detected)" after a manual pick from the detection-failed pickerPossibly taken @ShobhanKarthish claimed this 72 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
Maintainers usually reply within 1 day
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
Maintainers usually reply within 2 days
-
nip98: validateEvent skips the payload check for a string or bytes bodyMay be free again A pull request for this issue was closed without being merged. Open
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
nbd-wtf/nostr-tools#599 ·
Maintainers usually reply within 1 day
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
LemmyNet/lemmy-ui#4442 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Zettlr/Zettlr#6664 · 1 comment ·
Maintainers usually reply within 1 day
-
Microsoft Outlook add attachment reads the binary field from item 0Possibly taken @joshbarros claimed this today. Openstatus:in-linear
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
n8n-io/n8n#44021 · 2 comments ·
Maintainers usually reply within 1 day