Answering stack: one implementation for evaluate/benchmark/answers and arms/subjects

Start with dokimi/eval/arm_runner.py, dokimi/orchestrator/pipeline.py, and dokimi/eval/answerer.py, then read the listed CLI entry points and dokimi/records.py to trace how results reach consumers. The proposal requires characterization tests across both stacks before changing behavior; begin by reviewing the existing tests and the fake_llm and provider_calls fixtures in tests/conftest.py. Done means the pipeline delegates answering and judging to ArmRunner, preserves the answers.v1 and eval.v1 consumers through an adapter, and records method versions for changed results.

Split out of #94 (§5 row X), as @antonwolfy asked in his review: merging the two answering stacks is the largest refactor in the CLI plan, and "blocked on" hid its size. Facts below are from main @ 85b8d84a; paths are under dokimi/.

Problem

dokimi has two code paths that answer a question with an LLM, with and without docs, and judge the answers. They were written separately, and they give different numbers for the same question set.

	Stack P: EvaluationPipeline + Answerer	Stack A: ArmRunner
Core	orchestrator/pipeline.py:61 (answer :615, judge :647), eval/answerer.py:49	eval/arm_runner.py:123, treatments/arms.py
CLI	evaluate (commands/orchestrate.py:23), benchmark run (commands/benchmark.py:30, batch :222/:244/:300), answers generate (commands/answers.py:98, Answerer only)	arms run (commands/arms.py:82, _build_arms_output :606), arms matrix-run (:109), subjects run (commands/subjects.py:151)
Records	answers.v1 / eval.v1	arms.v1

No CI workflow runs either stack today, so the drift has gone unnoticed.

Differences that change scores

1. Retrieval request.
   • Stack P sends max_tokens=4000 and leaves max_results at its default (answerer.py:308). top_k is stored (:102) and never read; a TODO at :307 says so.
   • Stack A sends max_results=top_k and leaves max_tokens at the client default of 8000 (treatments/arms.py:81-83).
   • The clients cut context at max_tokens*4 characters, so that is 16k vs 32k. Context7's tokens parameter and cache key differ too (mcp/context7.py:119, :144), so the two stacks retrieve different docs for the same question.
2. No docs found.
   • Stack P judges the literal string "[FALLBACK: No relevant documentation found]" as the with-docs answer, with no model call (answerer.py:248-260).
   • Stack A makes a real call with the plain prompt, which is the baseline.
3. Rerank exception. Stack P falls back to the placeholder (answerer.py:337). Stack A uses the raw docs (treatments/arms.py:93-98).
4. Answer call.
   • Stack P calls llm_call_with_usage with no system prompt, max_tokens, temperature, seed or cache prefix (answerer.py:379, :417).
   • Stack A passes all five (arm_runner.py:377).
   • The prompts differ as well. Stack P puts the question first and says "use ONLY the documentation". Stack A puts the context first and says "where relevant".
5. What the judge sees.
   • Stack P shows 200-character Doc i: snippets (eval/judge.py:381, :606), and its baseline gets "(No documentation provided)".
   • Stack A shows the full material through eval/judge_context.py:278, stamped treatment_material.v1.
6. Judge temperature. benchmark does not pass it (commands/benchmark.py:52); evaluate does.
7. --max-tokens means opposite things. In Stack P it is the retrieval budget (benchmark.py:358). In Stack A it is the output ceiling (arms.py:270). This is the same drift that #94 §2.4 lists.
8. Library id. The pipeline resolves it without the registry (pipeline.py:622); answers generate and arms use the registry.
9. Statistics. Stack P reports unpaired means and deltas. Stack A reports paired tests with holm/bh and interactions.

Only in one stack

• Only in Stack A:
  • N arms, agentic arms, skills, sessions
  • --judge-panel
  • cost_summary (USD, cache hits, latency)
  • --grounded-from-topics
• Only in Stack P: personas, question generation and the question-set hash.
• Neither has: a budget cap, or resume. Only answers generate and eval score save incrementally.

Already shared

llm.py, the Judge class (Stack P calls evaluate_answers at judge.py:267; Stack A calls score_answer at :586), reranker.py, the MCP clients and factory, metrics/usage.py, records.py (it reads both shapes), save_artifact and defaults.py.

About ten consumers read answers.v1/eval.v1, including trust_gate, the dashboard, report, model_compare and ragas.

Dead code in Stack P

Answerer.llm (answerer.py:89-101) is never used. The empty-docs branch at :348-363 cannot be reached.

Drift is happening now

#214 adds a verbatim prompt style to both answerer.py and arm_runner.py, in two different shapes. #217 touches commands/arms.py, and #166 touches llm.py.

Why it blocks #94

#94 turns evaluate, benchmark and subjects into the pipeline, benchmark and subject recipes of plan/run (§2.2). With two stacks, a recipe would run on one stack while run arms runs on the other, and compare would put numbers side by side that were measured in different ways.

Proposal: keep Stack A

Stack A already covers Stack P's two arms as a special case: DocTreatment is with_docs and Baseline is without_docs. It also has paired statistics, cost accounting and panels. Stack P keeps what it does upstream of answering: personas, question generation and the answers.v1/eval.v1 consumers.

Steps. Each is its own PR, and a PR that changes numbers is never mixed with a refactor (#4 conventions):

1. Characterization test, no behaviour change. Run both stacks on one fixture and record, for each stack:

   • the retrieval arguments
   • the prompts
   • the provider call counts
   • the judge context
   • the scores
   • the cost

   It uses fake_llm and provider_calls (tests/conftest.py:53), plus a fake MCP client that records max_results/max_tokens, a stub reranker and frozen timestamps. Cases: docs found, no docs, rerank exception, a ground-truth chunk.

   The test pins today's differences as expected values. Later PRs flip them one at a time, so each change in numbers is visible in the diff.

   The tests that exist today do not cover this:

   • test_pipeline_question_set stubs whole pipeline steps.
   • test_eval_answerer patches llm_call_with_usage instead of using fake_llm.
   • test_provider_call_budgets counts calls but never compares the two stacks.

2. Fix Stack P in place. This changes numbers.

   • Honour top_k.
   • Pass the judge temperature in benchmark.
   • Resolve the library id through the registry.
   • Put placeholder judging behind an explicit flag, so a missing doc is no longer scored as an answer by default.
   • Delete the dead code.

   Stamp a method_version in the artifacts so that old and new runs are not compared silently.

3. One retrieval path. Stack P retrieves through DocTreatment. After this, the test from step 1 shows identical retrieval arguments.

4. The pipeline delegates answering and judging to ArmRunner.

   • Add an adapter from arms.v1 to answers.v1/eval.v1, so the ten consumers keep working.
   • Add incremental save to ArmRunner first. Without it, evaluate would lose the partial results it saves today.

5. Flag.

   • In the Stack P commands, --max-tokens becomes --retrieval-max-tokens, with a tombstone (#94 §2.4/F1b, concept "tokens").
   • In Stack A, --max-tokens keeps its meaning as the output ceiling.

6. Remove the rest.

   • The consumers read through records.py.
   • Answerer and the adapter are deprecated, then removed.
   • answers generate becomes run answers on the shared stack.

Before step 3, decide how #214 adds verbatim: either as one style in a shared prompt builder, or by landing it in Stack A only.

Risks

• Historical comparability. Steps 2–4 change scores, so old and new runs are not comparable. The method_version stamp lets compare refuse to put them side by side, or mark the comparison as exploratory.
• The trust gate. Its thresholds were calibrated on rows where the placeholder was judged. Re-check them after step 2.
• Old eval.v1 files have no judge-material stamp. records.py must read them as "material unknown".
• Scripts. The meaning of --max-tokens flips for existing scripts. The tombstone in step 5 has to catch the old use.
• Cache. The Context7 cache key changes, so the first runs after step 3 are slower and cost more.
• Complexity baselines. scripts/growth_baseline.json has entries for ArmRunner.judge and EvaluationPipeline.run. Update them in the PR that moves the code.

Out of scope

• The other duplicated subsystems: comparison statistics, artifact writers and so on. They are tracked in #94 §5.
• New features in ArmRunner.

Links: #94 (§2.2, §2.4, §5 X), #4 (waves 4 and 5), #121 (user loop), #214, #217, #166.