Track multi-route AI observability coverage and blocked evaluator recheck

Open
#1,353 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
35/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
typescript

Research direction

Start in services/pr-evaluator/evaluator.ts, focusing on how successive SDK success messages are handled and how the evaluator report is retained. Run pnpm test:evaluator, then rerun the updated rubric against the saved before/after diffs. Done means the complete report is preserved and the six route groups are checked for complete $ai_output_choices payloads, including role, plus the SSE variants and coverage report.

Written by the indexing model from the issue text.

Description

Status

Parked for now. This tracks the multi-route AI observability reproduction and the blocked before/after evaluation. No further runs are planned until work resumes.

Reproduction and findings

The Workbench fixture PR exercises six inference route groups: OpenAI-compatible chat, Responses, Ollama chat and generate, direct assistant tool calls, and WebSocket generation. It includes JSON, SSE, NDJSON, provider failures, and cancellation.

  • Before guidance changes, the Wizard run took 236.5 seconds and incorrectly captured a successful WebSocket generation as an error without output or tokens.
  • After applying Context Mill #406 and manually supplying the Markdown from docs #20409, the Wizard run took 352.5 seconds. The WebSocket classification improved, but five successful $ai_output_choices entries across four route groups still lack role. The promised inference coverage ledger/report was not written, and the SSE completion parser remains brittle to valid JSON whitespace.
  • Both runs added capture hooks for all six route groups and passed the fixture's five response-contract tests. Those checks do not establish that the emitted AI observability payloads are complete.

The earlier gateway evaluator scored both runs 5/5 despite the remaining payload/report misses. The Workbench PR adds specific rubric criteria and scoring caps; pnpm test:evaluator passed 57/57. The live recheck with that rubric did not yield a usable after score.

Evaluation blocker

In the live recheck, the evaluator logged a completed report result (10,784 output tokens), then another success result (28 output tokens). Its loop assigns resultText = message.result on each success, so the later result appears to replace the full report. It then retried and failed with No result received from agent (including retry). A model mismatch is possible but unconfirmed; the observed result replacement should be investigated first. This is in the Workbench evaluator (services/pr-evaluator/evaluator.ts), not established as the same failure as Wizard's long-running gateway connection issue #581.

When resumed

  1. Preserve the complete evaluator report when subsequent SDK result messages arrive, and rerun the updated rubric against the saved before/after diffs.
  2. Verify each successful generation has the required $ai_output_choices shape, including role, across all six routes; verify SSE variants and the promised coverage ledger/report.
  3. Decide whether any remaining misses need Context Mill guidance, docs, or Wizard behavior changes, using the Workbench fixture as the regression case.
Dominant language
TypeScript
Stars
204
Forks
54
Avg merge
1d 5h
Merged PRs (30d)
51

Getting set up

  • No Dockerfile or Docker Compose file
  • Has a pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from PostHog/wizard

All issues in PostHog/wizard

Similar issues

More TypeScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.