Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Operator limits: account the cost of runs that fail after using the model

Open
  • Is something wrong?
#273 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
22/100
Issue type
Feature
Clarity
Clearly specified
Activity status
Active
Tech stack
typescript
Domain
backend

Research direction

Start with the failure path of the Diff and Dynamic runners and the contract text for a failed result, where usage may already be set; the issue names no file paths, so locate them from there. The app's settle rule for failed results and the no_result reason are in the app layer, and the issue depends on #267, #272 and #274, so read those first. Done means cases T01 to T08 pass, with successful results unchanged.

Written by the indexing model from the issue text.

Description

area: app area: runner type: feature

Part of #268. Depends on #272 and #267.

Problem

A run reports its usage only when it succeeds. A run that fails after it used the model (a timeout at the end, a result the validator refuses, a publish error, a lost lease on the last attempt) still cost real money, but reports no usage. On a workspace that uses the operator's key this is spend nobody sees and nothing limits: a pull request that always fails near the end can be retried indefinitely at the operator's cost. The cost of a failed Dynamic run is the explicit out-of-scope item of #267.

Behavior

  • A run that fails after at least one model call reports the usage it has: the model proxy and the Diff run know their counters and (after #267 and #274) their cost at the time of the failure. The runner puts it in the failed result; the contract already has the shape (usage may be set on a failed result; this issue makes the runners fill it and the contract text say so).
  • Cost rules stay those of #267: all or nothing. If any call of the failed run is unpriced, the cost is unknown (null), the token counters are still reported.
  • The app settles the record of #272 from a failed result exactly like a successful one: with usage present it is settled (counts, cost possibly null); without any model call it is void.
  • A job that ends without any result from the runner (the runner died, the lease expired for the last time) stays settled with unknown cost, not void: the model may have been used, and counting it is the conservative reading. The record says so with a reason code (no_result).
  • Nothing about display changes: the Runs view keeps hiding usage of failed runs unless it already shows it.

Scope

Runner result building for the failure path (Diff and Dynamic), contract text and example, the app's settle rule for failed results and no_result, docs.

Out of scope

How the cost is used (#277), measuring Diff cost exactly (#274), retries and backoff policy.

Test cases

ID kind level case expected
T01 behavior unit (runner) a Dynamic run fails after three priced calls the failed result carries tokens and the summed cost
T02 behavior unit (runner) one of the calls is unpriced tokens reported, cost null
T03 behavior unit (runner) a Dynamic run fails before the first model call no usage, as today
T04 behavior unit (runner) a Diff run fails after its model call the usage PR-Agent measured is reported
T05 behavior integration (app) a failed result with usage record settled with that usage
T06 behavior integration (app) a job whose last lease expires with no result record settled, cost null, reason no_result
T07 contract unit the contract example for a failed result with usage valid in both suites
T08 regression unit successful results unchanged
Dominant language
TypeScript
Stars
0
Forks
1
Avg merge
1h 15m
Merged PRs (30d)
168

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from ExpidetGames/code-reviewer

All issues in ExpidetGames/code-reviewer

Similar issues

More TypeScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.