Distill a small rule-writer from recorded learning traces

Aperta
  • Qualcosa non va?
#10 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
35/100
Tipo di issue
Funzionalità
Chiarezza
Specificata chiaramente
Stato di attività
Tranquilla
Stack tecnologico
machine-learning, python
Ambito
machine-learning

Direzione di ricerca

Start with rulechef/training_logger.py to understand the trace format, then inspect benchmarks/benchmark_banking77.py for regenerating traces. Prepare filtered training data, fine-tune and serve a small model, and compare it with the teacher on a held-out task. Done means a reproducible data preparation script and training config, plus a comparison table reporting F1 and rule validity.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

enhancement

Goal

Train a small open model to do what a frontier LLM does in RuleChef — propose and patch rules — so rule-writing no longer requires a big model. This is the "specialized small model" direction and the most impactful contribution.

The data already exists

Every learning-time LLM call is logged as a ChatML (messages, response) pair by rulechef/training_logger.py, tagged with call_type. Format (see the module docstring):

{"messages": [{"role":"user","content":"<prompt>"}, {"role":"assistant","content":"<rules JSON>"}],
 "call_type": "rule_synthesis_per_class | rule_patch | guide_refinement | audit_rules | ...",
 "metadata": {...}, "timestamp": "..."}

Past runs produced ~500 such pairs across Banking77 / OntoNotes / Aegis. They're git-ignored (bulky) — regenerate by running the benchmarks with a TrainingDataLogger, or ask a maintainer for the existing *.training.jsonl. (Note: the TAB run's traces went to a temp dir and weren't kept — see #11; regenerate if you want TAB.)

What to do

  1. Assemble a fine-tuning set: concatenate the *.training.jsonl, keep only the messages field, filter by call_type (start with rule_synthesis_per_class).
  2. Fine-tune a small open model (e.g. 1–3B Qwen2.5 / Llama-3.2) on it.
  3. Serve it behind an OpenAI-compatible endpoint and pass it to RuleChef(client=..., model=...) — no core changes needed.
  4. Evaluate: learn rules with the small model vs the teacher on a held-out task; report F1 and rule validity rate.

Acceptance

A reproducible recipe (data prep script + training config) and a comparison table (small model vs teacher) on at least one task. Bonus: a patch-model (call_type=rule_patch).

Pointers

rulechef/training_logger.py (format + log()), benchmarks/benchmark_banking77.py / benchmark_extract.py (wire a TrainingDataLogger to regenerate traces), RuleChef(client=...) accepts any OpenAI-compatible client.

Start here

git clone https://github.com/KRLabsOrg/rulechef.git && cd rulechef
pip install -e ".[dev,grex]" && pytest tests/ -q   # should be green before you start

# inspect the training-trace format you will fine-tune on:
python -c "import rulechef.training_logger as t; print(t.__doc__)"
# regenerate traces: run benchmarks/benchmark_banking77.py with a TrainingDataLogger (see the script)
Lingua principale
Python
Stelle
34
Fork
9
Merge medio
9g 13h
PR unite (30g)
1

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di KRLabsOrg/rulechef

Tutte le issue di KRLabsOrg/rulechef

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.