Make transformation exact-match and failures field-aware

Aperta
  • Qualcosa non va?
#34 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
56/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Tranquilla
Stack tecnologico
python
Ambito
machine-learning

Direzione di ricerca

Start with rulechef/matching.py around lines 100–150 and rulechef/evaluation.py around lines 340–450, then inspect output_matcher and TRANSFORMATION references in rulechef and tests. Run the existing test suite with pytest -q to see the current behavior. Done means transformation outputs are compared by field, failures identify failed fields for default matching, custom matchers control correctness, and EvalResult.to_dict() retains failure details.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

correctness help wanted

Problem

evaluate_dataset() currently reuses entity matching for arbitrary transformation outputs. _get_entities() keeps only list-valued fields and discards their field names, so scalar fields are ignored entirely. Some list shapes are also mis-scored or crash because the downstream matcher expects entity dictionaries.

This means a structurally valid but completely wrong LLM-generated object can be reported as correct. For example, for a transformation whose expected output is:

{"company": "Acme", "amount": "5"}

an actual output of:

{"company": "Wrong", "amount": "999"}

currently produces exact_match=1.0, no failures, and zero TP/FP/FN. That prevents the evaluator from identifying the bad example for refinement.

Schema validity and output correctness are separate concerns: #33 handles fail-closed schema validation. This issue makes evaluation compare valid transformation values correctly.

Desired behavior

Add an explicit TaskType.TRANSFORMATION path to dataset evaluation instead of flattening transformation values into entities.

For the default comparator:

  • compare all top-level fields and report which fields failed;
  • distinguish a missing key from a present null value;
  • reject extra keys;
  • compare scalar values type-sensitively (1, 1.0, and True are different);
  • match present null only with present null;
  • keep list comparison order-independent while preserving duplicate multiplicity;
  • compare nested JSON values as field values, but keep diagnostics at the top-level field name for this first cut.

matching.outputs_match() already contains transformation comparison logic and Task.output_matcher already provides a custom boolean comparator. Reuse or refactor those paths rather than creating a second unrelated definition of transformation equality.

When Task.output_matcher is supplied, it controls document correctness. A custom boolean matcher cannot explain field-level differences, so failures should either omit failed_fields or mark those diagnostics as unavailable; do not present default-comparator differences as the reason a custom matcher failed.

Scope

  • Branch explicitly for transformations in rulechef/evaluation.py.
  • Make default transformation equality obey the semantics above in rulechef/matching.py.
  • Use the comparison result for document-level exact_match and normal refinement failures.
  • For default matching failures, include a bounded, sorted failed_fields list in the failure record; ensure EvalResult.to_dict() retains it.
  • Preserve all extraction, NER, and classification behavior.

Treat each top-level list as one field comparison in this issue. Do not reinterpret arbitrary arrays as extraction entities or invent TP/FP/FN denominators for structured JSON.

Acceptance tests

  • Correct scalar-only output is an exact match and produces no failure.
  • A wrong scalar reduces exact match and produces a failure naming that field.
  • Missing, extra, present-null, and type-mismatched fields are distinguished.
  • Wrong contents inside a list of objects are rejected.
  • Lists of scalar values do not crash.
  • List order is ignored but duplicate multiplicity is preserved.
  • Mixed scalar/list output fails when either field fails.
  • A custom matcher is called and controls document correctness.
  • EvalResult.to_dict() retains the new failure detail.
  • Existing extraction, NER, classification, and matching tests remain unchanged and green.

Non-goals

  • Recursive JSON-path diagnostics or semantic/LLM-as-judge comparison.
  • New micro/macro field metrics; micro_*, per_class, and TP/FP/FN remain entity-oriented and may stay zero for transformations.
  • Redesigning the learner's candidate-selection score for scalar-only transformation tasks. That should follow once exact-match behavior is trustworthy.
  • Changes to schema validation (#33).

Start here

git clone https://github.com/KRLabsOrg/rulechef.git
cd rulechef
pip install -e ".[dev]"
pytest -q

sed -n '100,150p' rulechef/matching.py
sed -n '340,450p' rulechef/evaluation.py
rg "output_matcher|TRANSFORMATION" rulechef tests
Lingua principale
Python
Stelle
34
Fork
9
Merge medio
9g 13h
PR unite (30g)
1

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di KRLabsOrg/rulechef

Tutte le issue di KRLabsOrg/rulechef

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.