Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Operator limits: exact model cost of Diff Reviews

Open
  • Is something wrong?
#274 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
12/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
python, typescript
Domain
backend

Research direction

Start in runner/.../pr_agent_runtime.py, where costUsd is currently summed from PR-Agent's own price table, and read the Dynamic Review accounting proxy from #267 that this must match. The first step is the design spike: choose one of the three listed mechanisms and record it in runner/README.md. Done means the cost is the exact sum of provider-reported costs, null when any call is unpriced, and tests T01 to T08 pass.

Written by the indexing model from the issue text.

Description

area: runner type: feature

Part of #268. Depends on #272 and #267.

Problem

#267 makes the cost of a Dynamic Review exact, because its model calls pass through the runner's accounting proxy and OpenRouter states the charge of every request. A Diff Review does not: PR-Agent calls the model itself through LiteLLM, and the costUsd it reports (runner/.../pr_agent_runtime.py) is a sum over the calls PR-Agent could price with its own price table, which can be missing, stale or different from the charge OpenRouter actually made. For a workspace that runs on the operator's key, Diff Review spend must be measured with the same exactness as Dynamic Review, or the usage records of #272 would be exact for one kind and approximate for the other.

Behavior

  • Decision first (spike inside this issue). Pick one mechanism, record it in the issue and runner/README.md, then implement it. Candidates, in order of preference:
    1. Route PR-Agent's OpenRouter calls through the same loopback accounting proxy the Dynamic Review uses (per-job capability, fixed destination, no agent-supplied routing), reading usage.cost of every answer, with #267's rules.
    2. Read the cost from LiteLLM's response object where OpenRouter's usage.cost is passed through, if it is the provider's value and not a table lookup.
    3. Ask OpenRouter for the generation record of each request id after the run (GET /api/v1/generation?id=…), if the ids are available and the lookup is cheap.
  • Whatever is chosen: the result's usage.costUsd is the exact sum of the provider-reported costs of all calls, all or nothing as in #267 (any unpriced call makes it null), never derived from a price table.
  • It applies to every Diff job, not only managed ones (a consistent shape; own-key jobs just do not use it for limits).
  • The request path still goes to OpenRouter's fixed address with the same credential as today; provider.require_parameters (#270) is unaffected.
  • If the chosen mechanism cannot be made exact for fallback-model retries and chunked reviews, the issue says exactly what is exact and what is null, and the tests pin it.

Scope

Runner Diff path, fakes (fake LLM already returns usage; extend it with cost), tests, runner/README.md, contract text for usage.costUsd.

Out of scope

The Dynamic path (#267), how cost is used, changing the review quality or prompts.

Test cases

ID kind level case expected
T01 behavior unit (runner) a Diff run with three model calls that each report a cost usage.costUsd is their exact decimal sum
T02 behavior unit (runner) one call has no cost costUsd is null, tokens still reported
T03 behavior unit (runner) fallback model is used after the first failed both attempts' costs are included, or the rule the spike set is applied and pinned
T04 behavior unit (runner) chunked review with several calls all calls counted
T05 regression integration (runner) Diff run against the fake LLM result unchanged except for the exact cost
T06 security unit the model destination and key are not controllable from the pull request or the model as before
T07 contract unit result validation in the runner and the app accepts the cost unchanged
T08 regression unit no price table or per-token rate used for the reported cost pinned by a test
Dominant language
TypeScript
Stars
0
Forks
1
Avg merge
1h 18m
Merged PRs (30d)
186

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from ExpidetGames/code-reviewer

All issues in ExpidetGames/code-reviewer

Similar issues

More TypeScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.