[feat] Show scorer results in the dashboard
Maintainers usually reply within 1 day
A pull request for this has already been merged.
- #1466 by @hz89Dev — merged
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 12/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- typescript
- Domain
- full-stack
Research direction
Start in dashboard.ts around line 228, where the server sets Access-Control-Allow-Origin, to see how the existing dashboard exposes its read API, then read docs/guide/dashboard.md and its zh-CN copy. The aggregation reads labels from teamai-reports and scores//.jsonl, which depend on #1178 and #1085, so confirm those shapes first. Done when acceptance criteria 1-8 pass as unit and API tests and the view renders from a fixture reports branch.
Written by the indexing model from the issue text.
Description
Parent: #1137
Goal
Labels on teamai-reports are only worth their cost if someone reads them. Add a scorer view to the existing teamai dashboard that answers two questions: is a scorer's pass rate moving, and does it differ by tool, model, team revision or definition.
What it shows
- One measurement at a time. Before any grouping, the view fixes the scorer, the rubric version (
rubric.sourceHash, withteamRevisionshown when there is one), the source (workfloworsession), the run mode (runMode, #1384) and the judge (judge.kind, and for a model judge its recorded tool and model). Labels that differ in any of these are never summed: a changed rubric, another kind of subject, another run mode or another judge is a different measurement. The view opens on the newest rubric whose labels carryrubric.teamRevision, so a member's uncommitted draft does not become the team's view. When no label of the scorer has one, it opens on the rubric with the latestscoredAt. Labels without judge data form their ownjudge not recordedcohort. - Pass rate over a window, 30 days by default, as a headline and as a weekly trend. The window is about when labels were scored, not when the work happened, and the view says so. A week with no labels is a gap, not 0%. The headline shows
n, passed and failed next to the rate. - Group by one recorded configuration dimension: tool, model, team revision, definition, the role, project or tag selection the assets were delivered for (
team.selection, #1178), or fingerprint (#1176). Session labels can also be grouped by a team skill the session loaded (used.skills, #1178), with a label counted under each skill it lists. Model groups keep whether the model was reported or only requested. Labels without that part form anot recordedgroup, never a guessed one. - Before and after a team commit. Given a commit of the team repo, the view splits the labels in two: those whose
agent.team.revisioncontains it, checked withgit merge-base --is-ancestorin this machine's team clone, and those that do not. A label withoutteam, or whose revision this clone does not have, isnot recorded. It combines with grouping by skill, so a team sees the sessions that loaded a skill before and after the commit that changed it. - Failing labels. Under a cohort, the view lists its failing labels: author,
scoredAt, platform andused.skills, newest first. When the team repo holds a pushed session whoseoriginis the label's subject, the row shows that session's name, so a teammate can open it withteamai session resumeor score it withteamai score --team-session. It shows no native session id, prompt or reasoning. - Pass/fail as stored.
passedis read from the label, never recomputed against the scorer's currentpassingScore. - A line under the view:
Labels from sessions and workflow runs show how the tools did on the work that came along. To compare tools, models or team assets on the same tasks, run a benchmark.
Data
- Workflow labels are read from
scores/<member>/<YYYY-MM>.jsonlonteamai-reports, like session labels, told apart bysource(#1085). For a workflow label,subject.runId,subject.stepandsubject.passjoin it to its run record onteamai-reports, which gives that pass'sagent. This view reads labels in #1085's shape. A label withoutrubric.sourceHashgoes to arubric not recordedcohort and is never counted under the current rubric. A label that names no pass, on a step that ran more than once, is shown under its step without a pass, and never attached to an arbitrary one. A record from before #1176, with a stringagent, gives only the definition. - Session labels come from
scores/*/*.jsonl(#1178). - Order. Read every source, keep the newest
scoredAtper(scorer, rubric.sourceHash, subject, judge.kind), and only then apply the window, so a label rescored in a later month counts once. - Benchmarks. Labels from a run whose end record has
benchmark(#1180) are left out. If #1180 lands later, it adds this exclusion. - Privacy. The dashboard server answers with
Access-Control-Allow-Origin: *(dashboard.ts:300). Its responses carry labels and recorded configuration fields only. No judge reasoning, prompt, transcript or diff is read or returned.
Scope
- Read API and page in the existing dashboard structure, next to the Workflow Runs page (#1088) if it has merged.
- English and Chinese strings.
docs/guide/dashboard.mdanddocs/guide/zh-CN/dashboard.md: a section on the scorer view and what it can and cannot show.
Acceptance criteria
- With fixture labels from two versions of a scorer, the view opens on the newest one whose labels carry
rubric.teamRevision, with its ownn, and the other shows only its labels. A version whose labels carry noteamRevisiondoes not open by default while another does. - Workflow and session labels of the same scorer are never in the same count. Neither are labels from two judge models with the same rubric.
- Grouping by model puts labels with no model under
not recorded, and keeps reported and requested-only models apart. - A workflow label is grouped by the
agentof the pass it judges. A label on a pre-#1176 record has only the definition. - A label scored in January and rescored in February counts once, in February.
- Raising
passingScorein the scorer file does not change any past pass rate. - A week without labels shows a gap in the trend.
- No API response contains a key or value from a judge's reasoning file or a session transcript (test with fixture text).
- Grouping by role puts labels from members with different
primaryRoleon the same team revision in different groups. Grouping by skill counts a label whoseused.skillsnames two skills under both. - A fixture benchmark run with
diff-qualitylabels leaves the view'snfordiff-qualityunchanged. - With labels on revisions
A,B(child ofA) andC(child ofB), splitting atBputsAbefore andBandCafter. A label with noteamisnot recorded. - A failing label whose subject is a pushed session is listed with that session's name and no native session id.
Test plan
- Unit tests for the aggregation: dedup before the window, cohorts, the join to passes, grouping, gaps.
- API tests for criterion 8.
- One real run of
teamai dashboardagainst a fixture reports branch, with a screenshot in the PR.
Out of scope
Benchmark comparisons in the dashboard, per-asset health (#408), judge reasoning in the browser, and alerts.
Dependencies
- Blocked by: #1178 and #1384. #1085 is transitive through #1178.
- Unblocks: #1249, #1269
Update: run modes and session labels
The original scope already covers session labels. This update adds the runMode split (#1384) and a reminder table. Run mode is fixed like source and judge: labels of different modes are never summed, and labels without runMode form their own not recorded cohort.
scorer runMode source labels passed
review-quality attended session 14 71%
review-quality unattended workflow 52 88%
review-quality not recorded session 9 67%
A member's own dashboard also compares the sessions the friction reminder fired on with those it did not, so the threshold can be tuned with data. "Reminder fired" is hinted in the member's local per-session contribute state (getSessionPath, src/contribute-check.ts:120-128), the learning offer. That state is only on the member's machine, so this table covers only sessions run on that machine, says so, and is not part of the team view.
| Column | What it counts |
|---|---|
| sessions | this machine's dashboard session_start events without launchedBy (#1384), in the window |
| labelled | those sessions with a label of this cohort, from the local label store and teamai-reports, deduplicated by #1178's key |
| passed | labelled sessions whose label passed; the rate is over labelled sessions only |
It compares within one scorer, rubric, source, judge and run mode. A session with no reminder state is not recorded; no reminder state is inferred from a label.
review-quality · rubric 3a9e1c · session · claude / <recorded-model> · attended · sessions run on this machine
reminder fired sessions labelled passed
yes 18 11 64%
no 96 40 85%
not recorded 12 3 —
The state file is swept 24 hours after its session's last write (cleanupStaleSessions, STALE_SESSION_MS, src/contribute-check.ts:183-208), which is shorter than the 30 days window. So the reminder decision must outlive it: when cleanup sweeps a state file, it keeps a record of only hinted, true or false, for every swept file (and the facts #1388 adds to it), removed after 30 days. #1388 needs the same record for its once-per-session offers; whichever of the two issues lands first adds it, and the other reuses it. The table stays local to this machine.
Added criteria:
- Labels of the same scorer with different
runModeare never in the same count; labels withoutrunModeare counted undernot recorded. - The reminder table reads only this machine's contribute state, and a label whose session has no state there is
not recorded. - A session whose reminder fired 10 days ago, whose state file was swept since, still counts under
yesin a 30 days window. - A session whose reminder did not fire, whose state file was swept 10 days ago, still counts under
noin a 30 days window.
- Dominant language
- TypeScript
- Stars
- 5.7k
- Forks
- 486
- Avg merge
- 13h 3m
- Merged PRs (30d)
- 474
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from Tencent/teamai-cli
-
Lint rule: require `windowsHide: true` on every child_process call in production sourcePossibly taken @SaulMoro claimed this today. Openenhancement help wanted
Difficulty 4/5 3-5 days Newbie friendliness 25/100
Tencent/teamai-cli#1763 · 2 comments ·
Maintainers usually reply within 1 day
-
enhancement help wanted
Difficulty 5/5 Over a week Newbie friendliness 38/100
Tencent/teamai-cli#1753 · 2 comments ·
Maintainers usually reply within 1 day
-
[bug] Stop-hook updater reports skipped or failed updates as ok and repeats unsupported-install warningsPossibly taken @YogeLiu claimed this 1 day ago. Open
Difficulty 4/5 3-5 days Newbie friendliness 35/100
Tencent/teamai-cli#1752 · 1 comment ·
Maintainers usually reply within 1 day
-
bug help wanted
Difficulty 5/5 Over a week Newbie friendliness 32/100
Tencent/teamai-cli#1743 · 2 comments ·
Maintainers usually reply within 1 day
-
enhancement help wanted
Difficulty 5/5 Over a week Newbie friendliness 25/100
Tencent/teamai-cli#1722 ·
Maintainers usually reply within 1 day
All issues in Tencent/teamai-cli
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day
-
binding:python bug
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
Maintainers usually reply within 1 day
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
coollabsio/orbit#72 ·
Maintainers usually reply within 1 day
-
Bug
Difficulty 2/5 1-3 hours Newbie friendliness 67/100
Maintainers usually reply within 1 day
-
bug needs-triage potential-regression
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
aws/aws-sdk-js-v3#8408 ·
Maintainers usually reply within 1 day