feat(scoring): support continuous 0–1 scores and named dimensions

Open
#317 15 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
12/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
go
Domain
devtools, testing

Research direction

This is design work spanning the scoring model, Agent judge contracts, script result parsing and reporting. Start from the existing boolean fields the issue names: AssertionResult.Passed, the judge summary.pass_rate, pass_threshold, and the all_required case aggregation, then settle the score contract with #192 and #246. Done means a documented, validated numeric contract with tests for partial credit, weighted totals, threshold boundaries, and gate failures.

Written by the indexing model from the issue text.

Description

enhancement

Problem

Evaluations currently expose boolean assertion/criterion decisions and derived pass rates, but cannot represent partial credit for a single criterion or native numerical scores across named quality dimensions.

For example, two outputs may both pass a case while differing substantially in correctness, completeness, and quality. Reporting only PASS/FAIL and case pass rates hides that improvement.

Current behavior:

  • AssertionResult.Passed and Agent judge criterion results are boolean.
  • Judge summary.pass_rate is the fraction of boolean assertions that pass; it is not an independently assigned continuous score.
  • Agent judging applies pass_threshold to that fraction.
  • Script judging maps exit code 0/nonzero to PASS/FAIL; stdout is evidence, not a parsed score contract.
  • Independent multi-judge results preserve judge identity, but case aggregation currently supports all_required, without numerical weighting.
  • Overall case pass rates and numerical quality scores need distinct names and semantics. Compatibility benchmark projections may use assertion/member pass rates and must not silently change their meaning.

Proposed capability

Add opt-in continuous scoring in the inclusive range [0, 1] and multiple named score dimensions, while retaining verdicts, evidence, and existing boolean evaluation behavior.

Named metrics are first-class evaluation results. A run must be able to report detailed metric scores without configuring weights or producing a single aggregate score. PASS/FAIL remains a separate verdict; a scalar total is an optional derived view.

Illustrative result shape when aggregate scoring is explicitly enabled (field names and exact nesting to be finalized):

{
  "status": "PASS",
  "score": 0.82,
  "scores": {
    "correctness": 0.9,
    "completeness": 0.8,
    "quality": 0.7
  }
}

The example total uses weights 0.4 / 0.4 / 0.2. Each dimension should preserve its judge/criterion provenance and supporting evidence; named dimensions need not map one-to-one to judges.

Named metrics as primary results
  • Support metrics-only results, for example scores: {"correctness": 0.9, "completeness": 0.75, "instruction_following": 1.0, "maintainability": 0.6}, with no required top-level score or weight configuration.
  • Each metric needs a stable ID, a definition/rubric, judge/criterion provenance, supporting evidence, and an explicit availability/error state. The compact example above is illustrative, not the complete metadata schema.
  • Allow independently configured thresholds for individual metrics when users want them to affect verdicts. Merely emitting a metric must not implicitly change PASS/FAIL behavior or require a common aggregate threshold.
  • Preserve and expose the original metric results even when optional aggregation is enabled. A total must not replace or hide improvements and regressions in individual dimensions.
Judge contracts
  • Support partial credit in Agent judge results through an explicit, program-owned scoring contract with stable criterion/dimension IDs and rubric definitions. Extend strict validation rather than accepting arbitrary model-produced score fields.
  • Add an opt-in structured numerical result contract for scripts, such as a dedicated JSON result file or explicit JSON output mode. Ordinary stdout must remain evidence for legacy scripts.
  • Keep existing rule-based and boolean-only configurations compatible. Any boolean-to-numerical conversion must be explicit and documented.
  • Reject malformed outputs, missing required scores, unknown/duplicate dimension IDs, out-of-range values, and non-finite numbers as ERROR rather than silently clamping or treating them as FAIL.
Aggregation and verdicts
  • Aggregate scores and weights are optional. When enabled, configure dimension/judge weights and score thresholds explicitly and calculate totals deterministically in skill-up.
  • Keep expect as hard gates, separate from score denominators. A failed gate must not be averaged away by high quality scores.
  • Preserve default all_required behavior; numerical aggregation is opt-in.
  • Define how score thresholds and any required judge/criterion verdicts interact.
  • Define ERROR, SKIP, timeout, and missing-dimension handling, including benchmark denominators. An unexecuted judge must remain distinguishable from a genuine score of zero.
  • Preserve judge identity, independent sessions, and per-member results; adding leaf criteria must not implicitly increase a judge's top-level weight. Current main uses a shared case runtime; per-judge runtime/filesystem isolation remains separately tracked in #308.
Reporting and comparisons
  • Persist named metric scores, evidence, and provenance in canonical machine-readable results and expose them individually in HTML/Markdown reports. Persist and display a total only when aggregation is enabled.
  • Report case pass rate, assertion pass rate, and numerical scores as distinct metrics.
  • Support per-metric means, standard deviations/distributions, and with-skill/without-skill or version-to-version deltas without requiring an aggregate total. Show valid sample counts and coverage for each metric so missing results cannot masquerade as genuine zero scores or improvements.
  • Compare only compatible metric definitions/rubrics; optional aggregate comparisons also require compatible weights. Define comparisons and denominators when cases have different dimension sets. Aggregate-total statistics remain optional.
  • Document compatibility projections for Anthropic grading/benchmark and JUnit, without silently redefining existing fields.
  • Review public API and evaluation-event schema compatibility where score data is exposed.

Acceptance criteria

  • An Agent judge can assign a criterion score such as 0.75 with traceable evidence.
  • A script can emit validated scores for at least two named dimensions.
  • Metrics-only evaluation reports at least two named metric scores with evidence and provenance, without requiring weights or an aggregate total.
  • Optional per-metric thresholds produce documented verdicts independently of aggregate scoring.
  • When aggregation is enabled, configured weights produce deterministic totals without replacing the underlying metric results.
  • Hard gates remain independent and cannot be compensated by weighted scores.
  • Missing/error/skipped scores have explicit representation and aggregation rules.
  • Reports and benchmarks retain individual metric scores alongside existing verdict/pass-rate fields; per-metric statistics, coverage, and compatible comparison deltas work without a total score.
  • Existing boolean-only configs and script exit-code behavior remain compatible.
  • Tests cover partial credit, weighted totals, threshold boundaries, invalid contracts, gate failures, mixed verdicts, and benchmark deltas.
  • English/Chinese documentation includes single-score and multi-dimension examples.

Related work and reference

  • #246 / PR #290 implemented independent multi-judge execution and reporting. Its completion notes explicitly defer numerical rewards, configurable weights/thresholds, and benchmark dimension metrics; this issue tracks that follow-up.
  • #300 separately tracks legacy gate/denominator correctness.
  • #192 covers strict Agent judge output contracts.
  • #308 separately tracks Docker/OpenSandbox multi-judge isolation.
  • Harbor supports named numerical rewards through reward.json: official task documentation. This is a capability reference, not a requirement to adopt Harbor's file paths or exact schema.

Runtime expansion, dependent judge pipelines, and replay/regrading are outside this scoring issue's scope.

Dominant language
Go
Stars
1.5k
Forks
134
Avg merge
1d 22h
Merged PRs (30d)
61

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from alibaba/skill-up

All issues in alibaba/skill-up

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.