Proposal: optional offline typed-decision judge for rubric scoring (SemIf / NanoJev pattern)

Inizia da replay_one() e dai percorsi esistenti backend.judge() e score_rule_judge(), quindi leggi skillopt_sleep/judges.py e tests/test_judges.py. La proposta identifica una possibile interfaccia score_typed_judge(), ma i maintainer devono ancora definire il confine dell’adattatore, lo standard delle evidenze e i criteri di accettazione prima che l’implementazione possa essere considerata completata.

What I am proposing

An optional judge backend that scores reference_kind="rubric" tasks with a typed-decision model - one that returns option probabilities in a single pass instead of generating text - rather than with a chat-model judge call. It would sit next to score_rule_judge and backend.judge(), and it should have an offline implementation, so none of it requires a hosted API, a vendor account or a key.

Why that seam

The judge contract in skillopt_sleep is (hard, soft, rationale), which is the shape a typed-decision model produces natively:

SkillOpt	typed decision
hard 0/1	the chosen option
soft 0..1	that option's probability, or a score head
rationale	the matched criteria labels

Insertion point is the else branch in replay_one() that calls backend.judge(task, response); rule and exact tasks keep scoring locally in score_rule_judge().

An established pattern, not speculation

• different-ai/openwork (23.6k stars) uses a typed model as a verification judge in its eval testkit for agent work.
• TheoLeeCJ/SemIf (1.7k stars, MIT, formerly OpenJev) states the same argument in its own README: most agent decisions are small - route this, retry that, does the evidence support X - and a chat model "spends time generating text", so it reads typed option probabilities directly, with no answer sentence, JSON repair, or decoding loop.
• notque/vexjoy-agent applies the same kind of model to request routing.

Why offline matters here

A judge that only works against a third-party hosted API adds a key, a vendor relationship and a network dependency to a path this project runs once per task, per candidate. An offline implementation avoids all three, and lets the judge live inside the existing no-network test rules - a fake client and a real one behind one interface. The interface should stay swappable regardless: the vendor's own system-one-adapter-python exists precisely to substitute chat models for comparison.

Candidate implementations

Repo	Stars	Licence	Notes
TheoLeeCJ/SemIf	1.7k	MIT	"Semantic ifs from open models", single 3090, plus a WebGPU browser demo; updated this week
vinnylarouge/jevlike	913	MIT	option-attention, one probability per option in one pass, in-repo checkpoints, CPU-runnable
TianyuCodings/NanoJev	487	MIT	0.6B, weights published on Hugging Face (I verified the files exist), no output-token decoding

I read their READMEs and repository metadata; I have not run any of them, so treat the fit as untested.

What this is not

This is an efficiency change, not a capability one. Skill quality is set by the optimizer that writes the edits and by how trustworthy the reward signal is; a cheaper judge changes how fast the loop measures, not whether it measures the right thing. Two consequences worth stating up front:

• The gate stays deterministic. Nothing here proposes putting a model decision inside evaluate_gate().
• Cheaper rubric judgement makes it more affordable to mine and gate more rubric tasks, and #154's evidence - raw check syntax reaching optimizer prompts, shallow proxies being Goodhart'd - is about exactly that failure mode. A cheaper oracle can make that mistake cheaper to commit. Agreement rate is the guard, and I do not have it yet.

A more interesting direction sits elsewhere: a sub-100ms typed model as an inference-time skill router rather than a judge would change what the project optimizes (per-task skill selection instead of one global skill). That is a different proposal and is not covered here.

Honest unknowns

Whether these decisions agree with the current judge on rubric tasks is the number that decides whether this is worth keeping at all, and I do not have it. rationale will also be thinner than a model's prose. The open implementations are weeks old and unaudited.

Minimal shape

• score_typed_judge() next to score_rule_judge() in skillopt_sleep/judges.py, tests in tests/test_judges.py.
• A small adapter so the decision model is swappable - local implementation first, hosted TypeSafe behind the same interface if anyone wants it.
• Optional extra; the core stays dependency-free; no-network tests with a fake client; an absent model or unreachable endpoint fails closed, matching the existing NO_AUTH behaviour.

Question for maintainers

Is there appetite for a third judge implementation at all - and if so, would an offline-only judge be preferable to one that calls a hosted model, and what evidence standard would you want before accepting one: agreement rate against the current judge on a fixed rubric set, paired A/B win/loss from the evalkit (#242), or something else?

Separately, and not dependent on any of the above: there is an official agent skill published alongside this model family. Evaluating it with a scenario pack like the ones in #254 would be the reverse direction and needs no judge work at all.