Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

Operator limits: account the cost of runs that fail after using the model

Aperta
  • Qualcosa non va?
#273 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
22/100
Tipo di issue
Funzionalità
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
typescript
Ambito
backend

Direzione di ricerca

Start with the failure path of the Diff and Dynamic runners and the contract text for a failed result, where usage may already be set; the issue names no file paths, so locate them from there. The app's settle rule for failed results and the no_result reason are in the app layer, and the issue depends on #267, #272 and #274, so read those first. Done means cases T01 to T08 pass, with successful results unchanged.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

area: app area: runner type: feature

Part of #268. Depends on #272 and #267.

Problem

A run reports its usage only when it succeeds. A run that fails after it used the model (a timeout at the end, a result the validator refuses, a publish error, a lost lease on the last attempt) still cost real money, but reports no usage. On a workspace that uses the operator's key this is spend nobody sees and nothing limits: a pull request that always fails near the end can be retried indefinitely at the operator's cost. The cost of a failed Dynamic run is the explicit out-of-scope item of #267.

Behavior

  • A run that fails after at least one model call reports the usage it has: the model proxy and the Diff run know their counters and (after #267 and #274) their cost at the time of the failure. The runner puts it in the failed result; the contract already has the shape (usage may be set on a failed result; this issue makes the runners fill it and the contract text say so).
  • Cost rules stay those of #267: all or nothing. If any call of the failed run is unpriced, the cost is unknown (null), the token counters are still reported.
  • The app settles the record of #272 from a failed result exactly like a successful one: with usage present it is settled (counts, cost possibly null); without any model call it is void.
  • A job that ends without any result from the runner (the runner died, the lease expired for the last time) stays settled with unknown cost, not void: the model may have been used, and counting it is the conservative reading. The record says so with a reason code (no_result).
  • Nothing about display changes: the Runs view keeps hiding usage of failed runs unless it already shows it.

Scope

Runner result building for the failure path (Diff and Dynamic), contract text and example, the app's settle rule for failed results and no_result, docs.

Out of scope

How the cost is used (#277), measuring Diff cost exactly (#274), retries and backoff policy.

Test cases

ID kind level case expected
T01 behavior unit (runner) a Dynamic run fails after three priced calls the failed result carries tokens and the summed cost
T02 behavior unit (runner) one of the calls is unpriced tokens reported, cost null
T03 behavior unit (runner) a Dynamic run fails before the first model call no usage, as today
T04 behavior unit (runner) a Diff run fails after its model call the usage PR-Agent measured is reported
T05 behavior integration (app) a failed result with usage record settled with that usage
T06 behavior integration (app) a job whose last lease expires with no result record settled, cost null, reason no_result
T07 contract unit the contract example for a failed result with usage valid in both suites
T08 regression unit successful results unchanged
Lingua principale
TypeScript
Stelle
0
Fork
1
Merge medio
1h 18m
PR unite (30g)
186

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di ExpidetGames/code-reviewer

Tutte le issue di ExpidetGames/code-reviewer

Issue simili

Altre issue su TypeScript

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.