feat(report): visualize evaluation history and comparison trends

Open
#349 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
go

Research direction

Start by reading the existing report artifact format and the comparison contract from #175; check how #317 represents named metrics and how reports are rendered. Then identify the current indexing, comparison, and report entry points in the repository. Done means retained runs can be indexed and compared locally with coverage, missing or failed states, configuration differences, and a rendered example, without rerunning evaluations.

Written by the indexing model from the issue text.

Description

Goal

Let Skill authors browse previous evaluation runs, track changes over time, and investigate improvements and regressions without manually opening individual result files.

Scope

  • Index completed and interrupted run artifacts using stable run IDs, timestamps, suite/case identity, Skill source revision or fingerprint, and resolved engine/model/runtime metadata.
  • Provide a local-first history view with filtering, baseline selection, and trend visualization for verdict/pass rate, available named metrics, token usage, and duration.
  • Show case-level changes and link each point or comparison to its original report and supporting artifacts.
  • Display missing data, failed runs, and metric coverage explicitly; distinguish execution errors from genuine zero scores.
  • Check comparability of datasets, judge/rubric definitions, and metric definitions. Surface configuration differences and incompatible comparisons.
  • Keep original results authoritative. A history index should be rebuildable from retained artifacts and must not require rerunning evaluations.
  • Define opt-in indexing, storage location, and retention behavior without exposing credentials or raw sensitive configuration.

The presentation may be an extended HTML report or another local viewer; a hosted dashboard/service is not required.

Acceptance criteria

  • Import/index at least three existing runs and browse them without calling an Agent or Judge.
  • Select a baseline and visualize run-level trends and case-level fixed/regressed/added/removed transitions.
  • Charts expose sample counts/coverage and retain explicit ERROR/SKIP/missing states.
  • Named metric trends work when #317 data is available, while legacy boolean-only reports remain usable.
  • Configuration changes and incompatible metric/rubric comparisons are visible.
  • Rebuilding an index preserves run identity and handles duplicates, missing artifacts, and malformed files clearly.
  • Regression tests cover indexing and comparison semantics, and a rendered example demonstrates the history view.

Related work

  • #175 owns offline comparison of two result files and CI gates; reuse that comparison contract where available.
  • #232 tracks automated evaluation of old versus new versions.
  • #317 defines continuous scores and named metrics.

This issue presents retained results; scheduling concurrent multi-version evaluations is tracked separately.

Dominant language
Go
Stars
1.5k
Forks
134
Avg merge
1d 22h
Merged PRs (30d)
61

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from alibaba/skill-up

All issues in alibaba/skill-up

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.