Allow per-stage model and reasoning selection

Open
#29 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
48/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
typescript

Research direction

No files, tests, or entry points are named. Start by locating the existing global model and reasoning configuration and the plan, packet review, verification, and composition stage dispatch. Done means stages can override either setting through the proposed configuration while unnamed stages retain the global defaults, with coverage for fallback behavior.

Written by the indexing model from the issue text.

Description

Problem

One model and one reasoning effort apply to every stage of a run. Stage 7 (packet review) dominates the token spend, so the strong model is paid for on work that does not need it.

Measured across runs of 0.5.6 on diffs of 9, 40 and 96 hunks (run.json / telemetry token counts):

stage share of total tokens
planning 1-2%
packet review 85-93%
verification 5-15%

Totals scale with the diff: ~0.9M tokens for 9 hunks, ~4.1M for 40, ~8M for 96. The Anthropic lane at opus/high reasoning uses about 1.7x the tokens of the openai-codex lane for the same diff.

Because the model is global, the only cost control available is to downgrade the packet review as well, which is the one stage where quality is the product.

Proposal

Allow the model and the reasoning effort to be selected per stage, falling back to the current global setting when a stage is not named:

[llm]
model = "..."            # default, unchanged

[llm.stages.plan]
model = "<cheaper model>"
reasoning = "low"

[llm.stages.verify]
reasoning = "medium"

Planning and composition look like the clear candidates for a cheaper model: they are structural work over text the pipeline already produced, and they are 1-2% and a few percent of the spend respectively. Verification is the interesting middle case - it is the main false-positive filter, so the value of a separate knob there is that it can be measured rather than assumed.

Why it matters

Per-stage selection turns "which model do I trust with this repository" into a per-stage question with a measurable answer, instead of one global decision that is paid for on every packet. It is also the prerequisite for anyone to publish a cost/quality curve for the tool.

Related but independent: #11 (grouping more related hunks) reduces the number of packet calls. The two levers compose - fewer calls, and the right model per call.

Happy to test a branch against a range of diff sizes and report tokens, wall time and finding deltas per stage.

Dominant language
TypeScript
Stars
8
Forks
1
Avg merge
5m
Merged PRs (30d)
2

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from 0xPolygon/codegenie

All issues in 0xPolygon/codegenie

Similar issues

More TypeScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.