feat(scoring): support continuous 0–1 scores and named dimensions
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 12/100
Research direction
This is design work spanning the scoring model, Agent judge contracts, script result parsing and reporting. Start from the existing boolean fields the issue names: AssertionResult.Passed, the judge summary.pass_rate, pass_threshold, and the all_required case aggregation, then settle the score contract with #192 and #246. Done means a documented, validated numeric contract with tests for partial credit, weighted totals, threshold boundaries, and gate failures.
Written by the indexing model from the issue text.
Description
Problem
Evaluations currently expose boolean assertion/criterion decisions and derived pass rates, but cannot represent partial credit for a single criterion or native numerical scores across named quality dimensions.
For example, two outputs may both pass a case while differing substantially in correctness, completeness, and quality. Reporting only PASS/FAIL and case pass rates hides that improvement.
Current behavior:
AssertionResult.Passedand Agent judge criterion results are boolean.- Judge
summary.pass_rateis the fraction of boolean assertions that pass; it is not an independently assigned continuous score. - Agent judging applies
pass_thresholdto that fraction. - Script judging maps exit code 0/nonzero to PASS/FAIL; stdout is evidence, not a parsed score contract.
- Independent multi-judge results preserve judge identity, but case aggregation currently supports
all_required, without numerical weighting. - Overall case pass rates and numerical quality scores need distinct names and semantics. Compatibility benchmark projections may use assertion/member pass rates and must not silently change their meaning.
Proposed capability
Add opt-in continuous scoring in the inclusive range [0, 1] and multiple named score dimensions, while retaining verdicts, evidence, and existing boolean evaluation behavior.
Named metrics are first-class evaluation results. A run must be able to report detailed metric scores without configuring weights or producing a single aggregate score. PASS/FAIL remains a separate verdict; a scalar total is an optional derived view.
Illustrative result shape when aggregate scoring is explicitly enabled (field names and exact nesting to be finalized):
{
"status": "PASS",
"score": 0.82,
"scores": {
"correctness": 0.9,
"completeness": 0.8,
"quality": 0.7
}
}
The example total uses weights 0.4 / 0.4 / 0.2. Each dimension should preserve its judge/criterion provenance and supporting evidence; named dimensions need not map one-to-one to judges.
Named metrics as primary results
- Support metrics-only results, for example
scores: {"correctness": 0.9, "completeness": 0.75, "instruction_following": 1.0, "maintainability": 0.6}, with no required top-levelscoreor weight configuration. - Each metric needs a stable ID, a definition/rubric, judge/criterion provenance, supporting evidence, and an explicit availability/error state. The compact example above is illustrative, not the complete metadata schema.
- Allow independently configured thresholds for individual metrics when users want them to affect verdicts. Merely emitting a metric must not implicitly change PASS/FAIL behavior or require a common aggregate threshold.
- Preserve and expose the original metric results even when optional aggregation is enabled. A total must not replace or hide improvements and regressions in individual dimensions.
Judge contracts
- Support partial credit in Agent judge results through an explicit, program-owned scoring contract with stable criterion/dimension IDs and rubric definitions. Extend strict validation rather than accepting arbitrary model-produced score fields.
- Add an opt-in structured numerical result contract for scripts, such as a dedicated JSON result file or explicit JSON output mode. Ordinary stdout must remain evidence for legacy scripts.
- Keep existing rule-based and boolean-only configurations compatible. Any boolean-to-numerical conversion must be explicit and documented.
- Reject malformed outputs, missing required scores, unknown/duplicate dimension IDs, out-of-range values, and non-finite numbers as ERROR rather than silently clamping or treating them as FAIL.
Aggregation and verdicts
- Aggregate scores and weights are optional. When enabled, configure dimension/judge weights and score thresholds explicitly and calculate totals deterministically in skill-up.
- Keep
expectas hard gates, separate from score denominators. A failed gate must not be averaged away by high quality scores. - Preserve default
all_requiredbehavior; numerical aggregation is opt-in. - Define how score thresholds and any required judge/criterion verdicts interact.
- Define ERROR, SKIP, timeout, and missing-dimension handling, including benchmark denominators. An unexecuted judge must remain distinguishable from a genuine score of zero.
- Preserve judge identity, independent sessions, and per-member results; adding leaf criteria must not implicitly increase a judge's top-level weight. Current main uses a shared case runtime; per-judge runtime/filesystem isolation remains separately tracked in #308.
Reporting and comparisons
- Persist named metric scores, evidence, and provenance in canonical machine-readable results and expose them individually in HTML/Markdown reports. Persist and display a total only when aggregation is enabled.
- Report case pass rate, assertion pass rate, and numerical scores as distinct metrics.
- Support per-metric means, standard deviations/distributions, and with-skill/without-skill or version-to-version deltas without requiring an aggregate total. Show valid sample counts and coverage for each metric so missing results cannot masquerade as genuine zero scores or improvements.
- Compare only compatible metric definitions/rubrics; optional aggregate comparisons also require compatible weights. Define comparisons and denominators when cases have different dimension sets. Aggregate-total statistics remain optional.
- Document compatibility projections for Anthropic grading/benchmark and JUnit, without silently redefining existing fields.
- Review public API and evaluation-event schema compatibility where score data is exposed.
Acceptance criteria
- An Agent judge can assign a criterion score such as 0.75 with traceable evidence.
- A script can emit validated scores for at least two named dimensions.
- Metrics-only evaluation reports at least two named metric scores with evidence and provenance, without requiring weights or an aggregate total.
- Optional per-metric thresholds produce documented verdicts independently of aggregate scoring.
- When aggregation is enabled, configured weights produce deterministic totals without replacing the underlying metric results.
- Hard gates remain independent and cannot be compensated by weighted scores.
- Missing/error/skipped scores have explicit representation and aggregation rules.
- Reports and benchmarks retain individual metric scores alongside existing verdict/pass-rate fields; per-metric statistics, coverage, and compatible comparison deltas work without a total score.
- Existing boolean-only configs and script exit-code behavior remain compatible.
- Tests cover partial credit, weighted totals, threshold boundaries, invalid contracts, gate failures, mixed verdicts, and benchmark deltas.
- English/Chinese documentation includes single-score and multi-dimension examples.
Related work and reference
- #246 / PR #290 implemented independent multi-judge execution and reporting. Its completion notes explicitly defer numerical rewards, configurable weights/thresholds, and benchmark dimension metrics; this issue tracks that follow-up.
- #300 separately tracks legacy gate/denominator correctness.
- #192 covers strict Agent judge output contracts.
- #308 separately tracks Docker/OpenSandbox multi-judge isolation.
- Harbor supports named numerical rewards through
reward.json: official task documentation. This is a capability reference, not a requirement to adopt Harbor's file paths or exact schema.
Runtime expansion, dependent judge pipelines, and replay/regrading are outside this scoring issue's scope.
- Dominant language
- Go
- Stars
- 1.5k
- Forks
- 134
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 61
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from alibaba/skill-up
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
Maintainers usually reply within 1 day
-
OpenSandbox runtime: expect.files_exist and file_contains fail because WorkspacePath is remotePossibly taken @bababoom721 claimed this 1 day ago. Open
Difficulty 4/5 3-5 days Newbie friendliness 25/100
Maintainers usually reply within 1 day
-
Difficulty 4/5 3-5 days Newbie friendliness 30/100
alibaba/skill-up#408 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 5/5 Over a week Newbie friendliness 15/100
alibaba/skill-up#407 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 4/5 3-5 days Newbie friendliness 25/100
alibaba/skill-up#406 · 1 comment ·
Maintainers usually reply within 1 day
All issues in alibaba/skill-up
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
OneBusAway/maglev#2145 ·
Maintainers usually reply within 2 days
-
priority/medium status/ready type/chore
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
discobox-ai/discobox#1357 ·
Maintainers usually reply within 1 day
-
TestPseudoLocaleRemote fails now and then: it types the code before the laptop asks for itPossibly taken @fzn0x claimed this today. Openbug priority: low
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
Maintainers usually reply within 1 day
-
edit_file diff rendering panics on invalid UTF-8 textPossibly taken @TDnorthgarden claimed this today. Openarea/tui
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
docker/docker-agent#4955 ·
Maintainers usually reply within 1 day
-
enhancement ui/ux
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
jbigler/railswitch#243 ·
Maintainers usually reply within 1 day