Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Architecture cleanup: fix live bugs, then deepen the LLM, eval-record, novelty, arms and process modules

Open
  • Is something wrong?
#4 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

@napetrov is already working on this.

Since Oct 22, 2026.

  • #164 by @napetrov — open
  • #166 by @napetrov — open
  • #172 by @napetrov — open

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
8/100
Issue type
Refactor
Clarity
Mostly clear
Activity status
Active
Tech stack
python
Domain
ai, tooling

Research direction

This is a multi-wave tracking issue, not a single task, so pick one checklist item and read its linked files first. The Wave 1 bugs in tasks run and eval panel-score are the most self-contained, with regression tests that should fail on main. Wave 4 files (llm.py, utils.py, eval/, report/) are a hold zone, so avoid them until the maintainers say otherwise. Done means the item's regression test fails before the change and passes after it.

Written by the indexing model from the issue text.

Description

enhancement

Why

An architecture review of dokimi found several concepts implemented in more than one place: the LLM call path, reading answer/eval records, spawning subprocesses, parsing arm specs, novelty coverage. The copies have drifted apart, and some of that drift is live bugs on main. This issue lists the fix series and the order it lands in, so reviewers can see the whole plan and contributors know which files will be busy.

Wave 1: bug fixes

Each PR carries a regression test that fails on main.

  • #8 tasks run with the default relative --output-dir can score a failed cell as passed. The agent writes its reward relative to the task checkout, the framework reads it relative to its own cwd, finds nothing, and falls back to returncode == 0.

  • #5 eval panel-score judges every answer with an empty question and empty context: it reads question/content, but the answerer writes question_text/snippet. Separately, the panel's parse retries replay the same cached reply.

  • #10 The trust gate's retrieval_hit_rate always passes, because nothing writes retrieval_hit. Fix: the judge and panel stamp it per evaluation, and older eval files report "not measured".

  • #11 The CLI offers openai-codex as a judge provider. Judge rejects it, but only after the answers have been paid for.

  • #24 Follow-up to #8: error the cell when its stale run dir can't be cleared.

Wave 2: current model defaults

  • #28 One default set on every surface: gpt-6.1-sol for answers and generation, claude-sonnet-5-5 for the judge. Drop gpt-4o-mini. Require litellm ≥ 1.105.0 for prices. Results will not be comparable with older baselines.

Wave 3: low-conflict refactors

  • #27 Write answers, eval and arms through artifacts.save_artifact (validated, atomic).
  • #22 Merge gate/soft_gate.py and gate/hard_gate.py.
  • #43 Add questions.v2 with a canonical question field; v1 text files are still read. Generated sets currently fail questions.v1.
  • #26 One arm-spec grammar module covering + and how names relate to specs.
  • #44 Novelty: the bundle record owns exclusion classes, coverage and the answer-key verdict (today the gate and the analyzer disagree on the unmeasured share). Follow-up: RunSettings and a thin CLI. Coverage basis: #101. settings: #105

Wave 4: central modules

Busy files during this wave: llm.py, utils.py, eval/*, report/*.

  • One LLM-call module for provider routing, retry, draws, cache and usage, replacing utils.get_llm. About 5 PRs. Step 1, the module with old entry points delegating: #78.
  • #80 eval/records.py: a single reader for answer, eval and arms records.
  • eval/compare.py: pairing, summaries and significance. This changes published numbers: report generate and report eval currently disagree on the same file.

Wave 5

  • Shared CLI model/provider flags, and an arms run module behind a typed run config instead of an argparse.Namespace hand-off.
  • #29 (git and misc sites; harnesses follow) One process runner: kills the whole process tree on timeout, decodes UTF-8, takes an explicit env, keeps partial output. Rebuilt on the current docker_solver. harnesses: #79 docker solver: #104

Wave 6

  • Design issue: a harness = a profile + where the runner runs + a transcript parser.

How these land

  • Each PR has one purpose. Fixes, behaviour-preserving refactors, and changes to numbers or defaults are never mixed in one PR.
  • At most two open PRs per hot area. The next one is rebased after the previous one merges.
  • Wave 4 lands back-to-back. Until it is through, please hold PRs that touch llm.py, eval/* or report/*, or mention them here.
Dominant language
Python
Stars
8
Forks
7
Avg merge
1d 1h
Merged PRs (30d)
167

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from intel/dokimi

All issues in intel/dokimi

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.