Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

Architecture cleanup: fix live bugs, then deepen the LLM, eval-record, novelty, arms and process modules

Aperta
  • Qualcosa non va?
#4 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
8/100
Tipo di issue
Refactoring
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
python
Ambito
ai, tooling

Direzione di ricerca

This is a multi-wave tracking issue, not a single task, so pick one checklist item and read its linked files first. The Wave 1 bugs in tasks run and eval panel-score are the most self-contained, with regression tests that should fail on main. Wave 4 files (llm.py, utils.py, eval/, report/) are a hold zone, so avoid them until the maintainers say otherwise. Done means the item's regression test fails before the change and passes after it.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

enhancement

Why

An architecture review of dokimi found several concepts implemented in more than one place: the LLM call path, reading answer/eval records, spawning subprocesses, parsing arm specs, novelty coverage. The copies have drifted apart, and some of that drift is live bugs on main. This issue lists the fix series and the order it lands in, so reviewers can see the whole plan and contributors know which files will be busy.

Wave 1: bug fixes

Each PR carries a regression test that fails on main.

  • #8 tasks run with the default relative --output-dir can score a failed cell as passed. The agent writes its reward relative to the task checkout, the framework reads it relative to its own cwd, finds nothing, and falls back to returncode == 0.

  • #5 eval panel-score judges every answer with an empty question and empty context: it reads question/content, but the answerer writes question_text/snippet. Separately, the panel's parse retries replay the same cached reply.

  • #10 The trust gate's retrieval_hit_rate always passes, because nothing writes retrieval_hit. Fix: the judge and panel stamp it per evaluation, and older eval files report "not measured".

  • #11 The CLI offers openai-codex as a judge provider. Judge rejects it, but only after the answers have been paid for.

  • #24 Follow-up to #8: error the cell when its stale run dir can't be cleared.

Wave 2: current model defaults

  • #28 One default set on every surface: gpt-6.1-sol for answers and generation, claude-sonnet-5-5 for the judge. Drop gpt-4o-mini. Require litellm ≥ 1.105.0 for prices. Results will not be comparable with older baselines.

Wave 3: low-conflict refactors

  • #27 Write answers, eval and arms through artifacts.save_artifact (validated, atomic).
  • #22 Merge gate/soft_gate.py and gate/hard_gate.py.
  • #43 Add questions.v2 with a canonical question field; v1 text files are still read. Generated sets currently fail questions.v1.
  • #26 One arm-spec grammar module covering + and how names relate to specs.
  • #44 Novelty: the bundle record owns exclusion classes, coverage and the answer-key verdict (today the gate and the analyzer disagree on the unmeasured share). Follow-up: RunSettings and a thin CLI. Coverage basis: #101. settings: #105

Wave 4: central modules

Busy files during this wave: llm.py, utils.py, eval/*, report/*.

  • One LLM-call module for provider routing, retry, draws, cache and usage, replacing utils.get_llm. About 5 PRs. Step 1, the module with old entry points delegating: #78.
  • #80 eval/records.py: a single reader for answer, eval and arms records.
  • eval/compare.py: pairing, summaries and significance. This changes published numbers: report generate and report eval currently disagree on the same file.

Wave 5

  • Shared CLI model/provider flags, and an arms run module behind a typed run config instead of an argparse.Namespace hand-off.
  • #29 (git and misc sites; harnesses follow) One process runner: kills the whole process tree on timeout, decodes UTF-8, takes an explicit env, keeps partial output. Rebuilt on the current docker_solver. harnesses: #79 docker solver: #104

Wave 6

  • Design issue: a harness = a profile + where the runner runs + a transcript parser.

How these land

  • Each PR has one purpose. Fixes, behaviour-preserving refactors, and changes to numbers or defaults are never mixed in one PR.
  • At most two open PRs per hot area. The next one is rebased after the previous one merges.
  • Wave 4 lands back-to-back. Until it is through, please hold PRs that touch llm.py, eval/* or report/*, or mention them here.
Lingua principale
Python
Stelle
8
Fork
7
Merge medio
1g 55m
PR unite (30g)
126

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di intel/dokimi

Tutte le issue di intel/dokimi

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.