Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

Proposal: optional offline typed-decision judge for rubric scoring (SemIf / NanoJev pattern)

Abierto
  • ¿Algo no está bien?
#283 1 comentario 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 6 días

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
28/100
Tipo de issue
Nueva funcionalidad
Claridad
Necesita aclaración
Estado de actividad
Activo
Stack tecnológico
python
Área
ai, backend

Línea de trabajo

Empieza por replay_one() y las rutas existentes de backend.judge() y score_rule_judge(), y después lee skillopt_sleep/judges.py y tests/test_judges.py. La propuesta identifica un posible punto de integración score_typed_judge(), pero los maintainers todavía deben definir el límite del adaptador, el estándar de evidencia y los criterios de aceptación antes de que la implementación pueda considerarse terminada.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

What I am proposing

An optional judge backend that scores reference_kind="rubric" tasks with a typed-decision model - one that returns option probabilities in a single pass instead of generating text - rather than with a chat-model judge call. It would sit next to score_rule_judge and backend.judge(), and it should have an offline implementation, so none of it requires a hosted API, a vendor account or a key.

Why that seam

The judge contract in skillopt_sleep is (hard, soft, rationale), which is the shape a typed-decision model produces natively:

SkillOpt typed decision
hard 0/1 the chosen option
soft 0..1 that option's probability, or a score head
rationale the matched criteria labels

Insertion point is the else branch in replay_one() that calls backend.judge(task, response); rule and exact tasks keep scoring locally in score_rule_judge().

An established pattern, not speculation

  • different-ai/openwork (23.6k stars) uses a typed model as a verification judge in its eval testkit for agent work.
  • TheoLeeCJ/SemIf (1.7k stars, MIT, formerly OpenJev) states the same argument in its own README: most agent decisions are small - route this, retry that, does the evidence support X - and a chat model "spends time generating text", so it reads typed option probabilities directly, with no answer sentence, JSON repair, or decoding loop.
  • notque/vexjoy-agent applies the same kind of model to request routing.

Why offline matters here

A judge that only works against a third-party hosted API adds a key, a vendor relationship and a network dependency to a path this project runs once per task, per candidate. An offline implementation avoids all three, and lets the judge live inside the existing no-network test rules - a fake client and a real one behind one interface. The interface should stay swappable regardless: the vendor's own system-one-adapter-python exists precisely to substitute chat models for comparison.

Candidate implementations

Repo Stars Licence Notes
TheoLeeCJ/SemIf 1.7k MIT "Semantic ifs from open models", single 3090, plus a WebGPU browser demo; updated this week
vinnylarouge/jevlike 913 MIT option-attention, one probability per option in one pass, in-repo checkpoints, CPU-runnable
TianyuCodings/NanoJev 487 MIT 0.6B, weights published on Hugging Face (I verified the files exist), no output-token decoding

I read their READMEs and repository metadata; I have not run any of them, so treat the fit as untested.

What this is not

This is an efficiency change, not a capability one. Skill quality is set by the optimizer that writes the edits and by how trustworthy the reward signal is; a cheaper judge changes how fast the loop measures, not whether it measures the right thing. Two consequences worth stating up front:

  • The gate stays deterministic. Nothing here proposes putting a model decision inside evaluate_gate().
  • Cheaper rubric judgement makes it more affordable to mine and gate more rubric tasks, and #154's evidence - raw check syntax reaching optimizer prompts, shallow proxies being Goodhart'd - is about exactly that failure mode. A cheaper oracle can make that mistake cheaper to commit. Agreement rate is the guard, and I do not have it yet.

A more interesting direction sits elsewhere: a sub-100ms typed model as an inference-time skill router rather than a judge would change what the project optimizes (per-task skill selection instead of one global skill). That is a different proposal and is not covered here.

Honest unknowns

Whether these decisions agree with the current judge on rubric tasks is the number that decides whether this is worth keeping at all, and I do not have it. rationale will also be thinner than a model's prose. The open implementations are weeks old and unaudited.

Minimal shape

  • score_typed_judge() next to score_rule_judge() in skillopt_sleep/judges.py, tests in tests/test_judges.py.
  • A small adapter so the decision model is swappable - local implementation first, hosted TypeSafe behind the same interface if anyone wants it.
  • Optional extra; the core stays dependency-free; no-network tests with a fake client; an absent model or unreachable endpoint fails closed, matching the existing NO_AUTH behaviour.

Question for maintainers

Is there appetite for a third judge implementation at all - and if so, would an offline-only judge be preferable to one that calls a hosted model, and what evidence standard would you want before accepting one: agreement rate against the current judge on a fixed rubric set, paired A/B win/loss from the evalkit (#242), or something else?

Separately, and not dependent on any of the above: there is an official agent skill published alongside this model family. Evaluating it with a scenario pack like the ones in #254 would be the reverse direction and needs no judge work at all.

Lenguaje dominante
Python
Estrellas
18.2k
Forks
1.7k
Merge medio
6 d 18 h
PR fusionados (30 d)
12

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de microsoft/SkillOpt

Todos los issues de microsoft/SkillOpt

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.