Calibration can pass invalid assessability data

Aperta
  • Qualcosa non va?
#649 0 commenti 0 reazioni 1 assegnatario Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

@Jonathas2127 ci sta già lavorando.

Dal 10/11/2026.

Valutazione

Questa issue non è ancora stata valutata.

Descrizione

Observed behavior

The offline calibration analyzer can return PASS for inputs that do not satisfy the assessability requirements in docs/rubric-calibration.md. I reproduced two cases on main at 5e00eab34335db7c9fcbf0e2212ca33a81d35908 using only the checked-in synthetic fixture:

  1. Replace every null model and human score in passing-synthetic.json with 82. The resulting corpus has no unassessable phases, but the analyzer still returns PASS with no failed gates.
  2. Keep the original corpus, and add a third rating to its first sample by copying reviewer-a, changing the reviewer ID, and setting only the third reviewer's Repeat score to null. The model and two reviewers score Repeat numerically, while the third reviewer marks it unassessable. The analyzer still returns PASS.

The control and both modified inputs also return exit 0 through the CLI in both JSON and Markdown formats. These are deliberately constructed synthetic counterexamples, not real interview data.

Expected behavior

The calibration contract requires complete and intentionally unassessable phases, says that reviewers use null when evidence cannot support a score, and treats a model/human disagreement about assessability as invalid.

Under the existing contract, a recorded human null is an explicit judgment that the phase is unassessable. It must remain part of assessability validation even when two or more other reviewers supplied numeric scores. A numeric model score paired with any human null is therefore an assessability mismatch and must be rejected. A corpus with no unassessable phase must fail coverage.

This concerns the offline calibration gate. The repository still declares the rubric NOT CALIBRATED; this reproduction does not establish a failure in a particular candidate's report or claim that real calibration has been completed.

Reproduction

From the repository root, run the following with Python 3.12.14 (the version I tested). It needs no credentials, model calls, or compiled server, and changes no tracked files.

Standalone reproduction: control, absent unassessable coverage, and one mixed human judgment 
python3 - <<'PY'
"""Run from any directory, passing the CodeTrial checkout as the argument."""

import copy
import importlib.util
import json
import subprocess
import sys
import tempfile
from pathlib import Path

root = Path(sys.argv[1]).resolve() if len(sys.argv) > 1 else Path.cwd()
script = root / "scripts/calibrate-framework.py"
spec = importlib.util.spec_from_file_location("calibration", script)
cal = importlib.util.module_from_spec(spec)
spec.loader.exec_module(cal)
baseline = json.loads(
    (root / "tests/fixtures/calibration/passing-synthetic.json").read_text(
        encoding="utf-8"
    )
)

no_unassessable = copy.deepcopy(baseline)
for sample in no_unassessable["samples"]:
    rows = [sample["modelScores"]] + [
        rating["scores"] for rating in sample["humanRatings"]
    ]
    for row in rows:
        for phase in cal.PHASES:
            if row[phase] is None:
                row[phase] = 82

mixed_assessability = copy.deepcopy(baseline)
third = copy.deepcopy(mixed_assessability["samples"][0]["humanRatings"][0])
third["reviewerId"] = "reviewer-c"
third["scores"]["Repeat"] = None
mixed_assessability["samples"][0]["humanRatings"].append(third)

cases = {
    "control": baseline,
    "no_unassessable": no_unassessable,
    "mixed_assessability": mixed_assessability,
}
results = []
with tempfile.TemporaryDirectory(prefix="calibration-issue-") as temporary:
    for name, raw in cases.items():
        input_path = Path(temporary) / f"{name}.json"
        input_path.write_text(json.dumps(raw), encoding="utf-8")
        for output_format in ("json", "markdown"):
            run = subprocess.run(
                [sys.executable, "-B", str(script), str(input_path), "--format", output_format],
                capture_output=True,
                text=True,
                check=False,
            )
            if output_format == "json":
                output = json.loads(run.stdout)
                status = output["status"]
                failures = output["failures"]
            else:
                status = run.stdout.splitlines()[0].split(": ", 1)[1]
                failures = []
            results.append(
                {
                    "case": name,
                    "format": output_format,
                    "exit": run.returncode,
                    "status": status,
                    "failures": failures,
                    "stderr": run.stderr,
                }
            )
assert all(item["status"] == "PASS" and item["exit"] == 0 for item in results)
print(json.dumps(results, indent=2))
PY

Observed output, shortened to the fields relevant here:

control               json      exit=0 status=PASS failures=[]
control               markdown  exit=0 status=PASS
no_unassessable        json      exit=0 status=PASS failures=[]
no_unassessable        markdown  exit=0 status=PASS
mixed_assessability   json      exit=0 status=PASS failures=[]
mixed_assessability   markdown  exit=0 status=PASS

Initial analysis

  • In analyze() at lines 176-184, only non-null human scores enter human[phase]. The later check at lines 185-196 detects a null model score paired with numeric human scores, or fewer than two numeric human scores for a numeric model score. Once two numeric ratings remain, an additional human null is no longer visible to validation.
  • The coverage gates beginning at line 212 track assessed phases, score bands and other strata, but do not track whether any consistently unassessable phase was present.

Proposed scope and acceptance checks

Keep this to offline assessability validation and its regression tests:

  • Retain every reviewer's null/numeric decision until phase consistency has been checked. Reject a numeric model score paired with any human null as an assessability mismatch, regardless of how many other numeric ratings are present. Apply the same consistency requirement when the model score is null. Keep the current requirement for at least two independent numeric ratings for an assessed phase.
  • Require at least one consistently unassessable (sample, phase) pair across the entire dataset. Count a pair only when the model score and every recorded human score for that phase are null, and the sample satisfies all existing input rules, including at least two distinct human reviewers and every required phase key. Multiple reviewers do not turn one pair into multiple examples. A structurally valid, assessability-consistent corpus with zero such pairs should return NOT_CALIBRATED, identify the missing coverage and exit 1. An assessability mismatch is invalid input and must exit 2 before coverage or metrics are evaluated.
  • A model-null/all-humans-null phase is valid as unassessable coverage and contributes no fabricated zero to numeric error or agreement metrics.
  • Add tests for two numeric raters plus one null, different reviewer orders, a null model with a numeric human, all-null agreement, the zero-versus-one unassessable-example boundary, and CLI JSON/Markdown statuses and exit codes. Missing reviewers, duplicate reviewer IDs and missing phase keys must remain invalid input rather than becoming qualifying examples. With all other gates satisfied, zero qualifying pairs must fail coverage and exactly one must satisfy this coverage check. The unchanged passing synthetic fixture should still pass, and generated fixtures must remain current.
  • Keep the existing thresholds, numeric metric formulas and input format. Document the one-pair minimum and mismatch rejection behavior in docs/rubric-calibration.md. No live scoring, prompt, report-schema or UI change is proposed.

The minimum coverage rule is dataset-wide: at least one consistently unassessable (sample, phase) pair. It does not require one example for every phase, session, language or difficulty. This is the minimum coverage check for this fix, not a claim that one example is statistically sufficient for calibration; all existing sample-size, stratum and metric gates still apply.

Acceptance examples below assume all unrelated coverage and metric gates are satisfied:

Input Expected result
Valid and assessability-consistent, with zero qualifying pairs NOT_CALIBRATED, exit 1, naming the missing unassessable coverage
Valid and assessability-consistent, with exactly one qualifying pair PASS, exit 0
Numeric model score, two numeric human scores and one human null Invalid input, exit 2
Null model score and any numeric human score Invalid input, exit 2

I plan to implement this bounded scope after maintainer review and assignment. The fix enforces the existing meaning of null as an unassessable judgment and adds the missing corpus-level coverage check.

Validation and related work

  • python3 -m unittest discover -s tests -p test_calibrate_framework.py -v: all 8 existing tests pass.
  • python3 scripts/gen-calibration-fixtures.py --check: passes.
  • The reproduction above passes its assertions for both counterexamples and the unchanged control.
  • Tested on Windows with Python 3.12.14. No complete Rust/browser gate or real human-rating study was run for this draft.
  • I searched open and closed issues/PRs for calibration and the analyzer's name. I found no report of these two validation gaps in the reviewed results. PR #189 concerns displaying phase scores, while PR #415 concerns synthetic report stability; this proposal concerns the offline human-rating analyzer.
Lingua principale
Rust
Stelle
228
Fork
150
Merge medio
1g 3h
PR unite (30g)
230

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di sysprog21/codetrial

Tutte le issue di sysprog21/codetrial

Issue simili

Altre issue su Rust

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.