feat(report): visualize evaluation history and comparison trends

Aperta
#349 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
35/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
go

Direzione di ricerca

Start by reading the existing report artifact format and the comparison contract from #175; check how #317 represents named metrics and how reports are rendered. Then identify the current indexing, comparison, and report entry points in the repository. Done means retained runs can be indexed and compared locally with coverage, missing or failed states, configuration differences, and a rendered example, without rerunning evaluations.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Goal

Let Skill authors browse previous evaluation runs, track changes over time, and investigate improvements and regressions without manually opening individual result files.

Scope

  • Index completed and interrupted run artifacts using stable run IDs, timestamps, suite/case identity, Skill source revision or fingerprint, and resolved engine/model/runtime metadata.
  • Provide a local-first history view with filtering, baseline selection, and trend visualization for verdict/pass rate, available named metrics, token usage, and duration.
  • Show case-level changes and link each point or comparison to its original report and supporting artifacts.
  • Display missing data, failed runs, and metric coverage explicitly; distinguish execution errors from genuine zero scores.
  • Check comparability of datasets, judge/rubric definitions, and metric definitions. Surface configuration differences and incompatible comparisons.
  • Keep original results authoritative. A history index should be rebuildable from retained artifacts and must not require rerunning evaluations.
  • Define opt-in indexing, storage location, and retention behavior without exposing credentials or raw sensitive configuration.

The presentation may be an extended HTML report or another local viewer; a hosted dashboard/service is not required.

Acceptance criteria

  • Import/index at least three existing runs and browse them without calling an Agent or Judge.
  • Select a baseline and visualize run-level trends and case-level fixed/regressed/added/removed transitions.
  • Charts expose sample counts/coverage and retain explicit ERROR/SKIP/missing states.
  • Named metric trends work when #317 data is available, while legacy boolean-only reports remain usable.
  • Configuration changes and incompatible metric/rubric comparisons are visible.
  • Rebuilding an index preserves run identity and handles duplicates, missing artifacts, and malformed files clearly.
  • Regression tests cover indexing and comparison semantics, and a rendered example demonstrates the history view.

Related work

  • #175 owns offline comparison of two result files and CI gates; reuse that comparison contract where available.
  • #232 tracks automated evaluation of old versus new versions.
  • #317 defines continuous scores and named metrics.

This issue presents retained results; scheduling concurrent multi-version evaluations is tracked separately.

Lingua principale
Go
Stelle
1.5k
Fork
134
Merge medio
1g 22h
PR unite (30g)
61

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di alibaba/skill-up

Tutte le issue di alibaba/skill-up

Issue simili

Altre issue su Go

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.