[feat] Show scorer results in the dashboard

Closed
#1,182 6 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

A pull request for this has already been merged.

  • #1466 by @hz89Dev — merged

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
12/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
typescript
Domain
full-stack

Research direction

Start in dashboard.ts around line 228, where the server sets Access-Control-Allow-Origin, to see how the existing dashboard exposes its read API, then read docs/guide/dashboard.md and its zh-CN copy. The aggregation reads labels from teamai-reports and scores//.jsonl, which depend on #1178 and #1085, so confirm those shapes first. Done when acceptance criteria 1-8 pass as unit and API tests and the view renders from a fixture reports branch.

Written by the indexing model from the issue text.

Description

enhancement

Parent: #1137

Goal

Labels on teamai-reports are only worth their cost if someone reads them. Add a scorer view to the existing teamai dashboard that answers two questions: is a scorer's pass rate moving, and does it differ by tool, model, team revision or definition.

What it shows

  • One measurement at a time. Before any grouping, the view fixes the scorer, the rubric version (rubric.sourceHash, with teamRevision shown when there is one), the source (workflow or session), the run mode (runMode, #1384) and the judge (judge.kind, and for a model judge its recorded tool and model). Labels that differ in any of these are never summed: a changed rubric, another kind of subject, another run mode or another judge is a different measurement. The view opens on the newest rubric whose labels carry rubric.teamRevision, so a member's uncommitted draft does not become the team's view. When no label of the scorer has one, it opens on the rubric with the latest scoredAt. Labels without judge data form their own judge not recorded cohort.
  • Pass rate over a window, 30 days by default, as a headline and as a weekly trend. The window is about when labels were scored, not when the work happened, and the view says so. A week with no labels is a gap, not 0%. The headline shows n, passed and failed next to the rate.
  • Group by one recorded configuration dimension: tool, model, team revision, definition, the role, project or tag selection the assets were delivered for (team.selection, #1178), or fingerprint (#1176). Session labels can also be grouped by a team skill the session loaded (used.skills, #1178), with a label counted under each skill it lists. Model groups keep whether the model was reported or only requested. Labels without that part form a not recorded group, never a guessed one.
  • Before and after a team commit. Given a commit of the team repo, the view splits the labels in two: those whose agent.team.revision contains it, checked with git merge-base --is-ancestor in this machine's team clone, and those that do not. A label without team, or whose revision this clone does not have, is not recorded. It combines with grouping by skill, so a team sees the sessions that loaded a skill before and after the commit that changed it.
  • Failing labels. Under a cohort, the view lists its failing labels: author, scoredAt, platform and used.skills, newest first. When the team repo holds a pushed session whose origin is the label's subject, the row shows that session's name, so a teammate can open it with teamai session resume or score it with teamai score --team-session. It shows no native session id, prompt or reasoning.
  • Pass/fail as stored. passed is read from the label, never recomputed against the scorer's current passingScore.
  • A line under the view: Labels from sessions and workflow runs show how the tools did on the work that came along. To compare tools, models or team assets on the same tasks, run a benchmark.

Data

  • Workflow labels are read from scores/<member>/<YYYY-MM>.jsonl on teamai-reports, like session labels, told apart by source (#1085). For a workflow label, subject.runId, subject.step and subject.pass join it to its run record on teamai-reports, which gives that pass's agent. This view reads labels in #1085's shape. A label without rubric.sourceHash goes to a rubric not recorded cohort and is never counted under the current rubric. A label that names no pass, on a step that ran more than once, is shown under its step without a pass, and never attached to an arbitrary one. A record from before #1176, with a string agent, gives only the definition.
  • Session labels come from scores/*/*.jsonl (#1178).
  • Order. Read every source, keep the newest scoredAt per (scorer, rubric.sourceHash, subject, judge.kind), and only then apply the window, so a label rescored in a later month counts once.
  • Benchmarks. Labels from a run whose end record has benchmark (#1180) are left out. If #1180 lands later, it adds this exclusion.
  • Privacy. The dashboard server answers with Access-Control-Allow-Origin: * (dashboard.ts:300). Its responses carry labels and recorded configuration fields only. No judge reasoning, prompt, transcript or diff is read or returned.

Scope

  • Read API and page in the existing dashboard structure, next to the Workflow Runs page (#1088) if it has merged.
  • English and Chinese strings.
  • docs/guide/dashboard.md and docs/guide/zh-CN/dashboard.md: a section on the scorer view and what it can and cannot show.

Acceptance criteria

  1. With fixture labels from two versions of a scorer, the view opens on the newest one whose labels carry rubric.teamRevision, with its own n, and the other shows only its labels. A version whose labels carry no teamRevision does not open by default while another does.
  2. Workflow and session labels of the same scorer are never in the same count. Neither are labels from two judge models with the same rubric.
  3. Grouping by model puts labels with no model under not recorded, and keeps reported and requested-only models apart.
  4. A workflow label is grouped by the agent of the pass it judges. A label on a pre-#1176 record has only the definition.
  5. A label scored in January and rescored in February counts once, in February.
  6. Raising passingScore in the scorer file does not change any past pass rate.
  7. A week without labels shows a gap in the trend.
  8. No API response contains a key or value from a judge's reasoning file or a session transcript (test with fixture text).
  9. Grouping by role puts labels from members with different primaryRole on the same team revision in different groups. Grouping by skill counts a label whose used.skills names two skills under both.
  10. A fixture benchmark run with diff-quality labels leaves the view's n for diff-quality unchanged.
  11. With labels on revisions A, B (child of A) and C (child of B), splitting at B puts A before and B and C after. A label with no team is not recorded.
  12. A failing label whose subject is a pushed session is listed with that session's name and no native session id.

Test plan

  • Unit tests for the aggregation: dedup before the window, cohorts, the join to passes, grouping, gaps.
  • API tests for criterion 8.
  • One real run of teamai dashboard against a fixture reports branch, with a screenshot in the PR.

Out of scope

Benchmark comparisons in the dashboard, per-asset health (#408), judge reasoning in the browser, and alerts.

Dependencies

  • Blocked by: #1178 and #1384. #1085 is transitive through #1178.
  • Unblocks: #1249, #1269

Update: run modes and session labels

The original scope already covers session labels. This update adds the runMode split (#1384) and a reminder table. Run mode is fixed like source and judge: labels of different modes are never summed, and labels without runMode form their own not recorded cohort.

scorer           runMode       source    labels   passed
review-quality   attended      session   14       71%
review-quality   unattended    workflow  52       88%
review-quality   not recorded  session    9       67%

A member's own dashboard also compares the sessions the friction reminder fired on with those it did not, so the threshold can be tuned with data. "Reminder fired" is hinted in the member's local per-session contribute state (getSessionPath, src/contribute-check.ts:120-128), the learning offer. That state is only on the member's machine, so this table covers only sessions run on that machine, says so, and is not part of the team view.

Column What it counts
sessions this machine's dashboard session_start events without launchedBy (#1384), in the window
labelled those sessions with a label of this cohort, from the local label store and teamai-reports, deduplicated by #1178's key
passed labelled sessions whose label passed; the rate is over labelled sessions only

It compares within one scorer, rubric, source, judge and run mode. A session with no reminder state is not recorded; no reminder state is inferred from a label.

review-quality · rubric 3a9e1c · session · claude / <recorded-model> · attended · sessions run on this machine
reminder fired   sessions   labelled   passed
yes              18         11         64%
no               96         40         85%
not recorded     12          3         —

The state file is swept 24 hours after its session's last write (cleanupStaleSessions, STALE_SESSION_MS, src/contribute-check.ts:183-208), which is shorter than the 30 days window. So the reminder decision must outlive it: when cleanup sweeps a state file, it keeps a record of only hinted, true or false, for every swept file (and the facts #1388 adds to it), removed after 30 days. #1388 needs the same record for its once-per-session offers; whichever of the two issues lands first adds it, and the other reuses it. The table stays local to this machine.

Added criteria:

  1. Labels of the same scorer with different runMode are never in the same count; labels without runMode are counted under not recorded.
  2. The reminder table reads only this machine's contribute state, and a label whose session has no state there is not recorded.
  3. A session whose reminder fired 10 days ago, whose state file was swept since, still counts under yes in a 30 days window.
  4. A session whose reminder did not fire, whose state file was swept 10 days ago, still counts under no in a 30 days window.
Dominant language
TypeScript
Stars
5.7k
Forks
486
Avg merge
13h 3m
Merged PRs (30d)
474

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from Tencent/teamai-cli

All issues in Tencent/teamai-cli

Similar issues

More TypeScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.