Operator limits: record usage per run with its measured cost
I maintainer di solito rispondono entro 1 giorno
Valutazione
- Difficoltà
- 4/5
- Tempo stimato
- 3-5 giorni
- Idoneità per principianti
- 18/100
- Tipo di issue
- Funzionalità
- Chiarezza
- Specificata chiaramente
- Stato di attività
- Ferma
- Stack tecnologico
- postgresql, sqlite, typescript
Direzione di ricerca
Start in the job claim and complete paths in src/jobs, where the reserve and settle inserts go, and read src/jobs/README.md and src/runner-api/README.md. Add the usage_records table with its migration and a scoped repository, then cover it in scoping.test.ts and scoped-isolation.integration.test.ts. Done when T01 to T10 pass on both SQLite and Postgres. A linked PR (#297) is already open and the issue depends on #267, so check with maintainers before starting.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Descrizione
Part of #268. Depends on #267 (the exact model cost of Dynamic Review runs): this issue stores the cost that issue reports and does not measure anything itself.
Problem
A job's usage (tokens, costUsd) is stored inside the JSON result of the job. That is enough to show one run, but not to ask "how many Diff and Dynamic reviews did this workspace run since X" or "how much did the operator's key spend", atomically with the decision to start a run, and in a form an external service can read incrementally. Limits (#277), the usage page (#282) and the operator usage API (#278) all need that.
Behavior
- New table
usage_records, one row per job (uniquejob_id), scoped to the workspace:id(monotonic),workspace_id,job_id,review_kind(diff/dynamic),key_mode(ownwhen the job ran on the workspace's key,managedwhen it ran on the operator's key, i.e. thesourceof the resolved LLM access),model_tier(nullable, filled by #280),state(reserved,settled,void),cost_micro_usd(nullable integer: the job'susage.costUsdtimes one million, rounded half up; null when unknown),prompt_tokens,completion_tokens(nullable),reserved_at,settled_at(null until settled). - Reserve. When the claim hands a job to a runner, the record is inserted in the same transaction as the claim (state
reserved). A job that is claimed again after a lost lease reuses its record; it never counts twice. - Settle. When the job completes, the record is settled from the result: with a reported usage it is
settledwith its cost (possibly null) and tokens; a result with no model call made (no usage and the job failed before the model was used: checkout, configuration, lease loss that ends in a failure, cancellation) turns the recordvoid. A record is immutable oncesettledorvoid. voidandreservedrecords exist for the counting rules of #277:reservedandsettledcount,voiddoes not.- Reading is by repository functions only: count of records by kind since a time, sum of settled managed cost since a time, a page of settled records ordered by
(settled_at, id)after a cursor. - Records are small and kept; no retention job in this issue.
- A run on a workspace without any entitlement is recorded as well (key mode from the access source), so that usage exists the moment limits are switched on. This does not change what the run does.
Scope
Table, migration, repository (scoped, classified in scoping.test.ts, rows in scoped-isolation.integration.test.ts), the insert in the claim transaction, the settle in the complete path, tests, docs in src/jobs/README.md and src/runner-api/README.md.
Out of scope
Limits and refusal (#277), runs that fail after using the model (#273), the Diff cost (#274), any endpoint (#278), UI (#282).
Depends on
#267. Can run in parallel with #271.
Test cases
| ID | kind | level | case | expected |
|---|---|---|---|---|
| T01 | behavior | integration | a job is claimed | one reserved record in the same transaction, with kind and key mode of the job |
| T02 | behavior | integration | the job completes with usage.costUsd = 0.0123456 |
record settled, cost_micro_usd = 12346, tokens stored |
| T03 | behavior | integration | the result has costUsd: null |
settled with null cost, not void |
| T04 | behavior | integration | the job fails before any model call | record void |
| T05 | behavior | integration | the lease is lost and the job is claimed again | the same record, still one |
| T06 | security | integration | repository of workspace A | cannot see or settle B's records (scoping tests) |
| T07 | behavior | integration | complete is repeated (alreadyCompleted) | the record is not changed again |
| T08 | regression | integration | a workspace without entitlements | runs exactly as before and still gets a record |
| T09 | behavior | integration | page of settled records after a cursor | ordered by (settled_at, id), stable under inserts, never contains reserved or void |
| T10 | regression | integration | SQLite and Postgres | same shape (parity test) |
- Lingua principale
- TypeScript
- Stelle
- 0
- Fork
- 1
- Merge medio
- 1h 18m
- PR unite (30g)
- 186
Preparare l'ambiente
- Include un Dockerfile o un file Docker Compose
- Ha un modello di pull request
- Leggi la guida per i contributori
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Altre issue di ExpidetGames/code-reviewer
-
area: app type: bug
Difficoltà 2/5 1-3 ore Idoneità per principianti 70/100
ExpidetGames/code-reviewer#308 ·
I maintainer di solito rispondono entro 1 giorno
-
type: docs
Difficoltà 2/5 1-3 ore Idoneità per principianti 78/100
ExpidetGames/code-reviewer#94 ·
I maintainer di solito rispondono entro 1 giorno
-
area: app post-mvp type: feature
Difficoltà 5/5 Più di una settimana Idoneità per principianti 12/100
ExpidetGames/code-reviewer#339 ·
I maintainer di solito rispondono entro 1 giorno
-
area: runner post-mvp type: feature
Difficoltà 5/5 Più di una settimana Idoneità per principianti 18/100
ExpidetGames/code-reviewer#338 ·
I maintainer di solito rispondono entro 1 giorno
-
area: runner post-mvp type: feature
Difficoltà 5/5 Più di una settimana Idoneità per principianti 12/100
ExpidetGames/code-reviewer#337 ·
I maintainer di solito rispondono entro 1 giorno
Tutte le issue di ExpidetGames/code-reviewer
Issue simili
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 68/100
apache/rocketmq-dashboard#6872 ·
I maintainer di solito rispondono entro 3 giorni
-
bug
Difficoltà 2/5 1-3 ore Idoneità per principianti 70/100
fab-geocommuns/RNB-site#539 ·
I maintainer di solito rispondono entro 2 giorni
-
Guest multiview tile label leaks raw virtual-source-id "Whip" instead of the guest name / slot labelAperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 62/100
Eyevinn/open-live-studio#211 ·
I maintainer di solito rispondono entro 1 giorno
-
documentation good first issue
Difficoltà 2/5 Mezza giornata Idoneità per principianti 72/100
sujithputta02/Esapay#13 ·
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 65/100
ehmpathy/rhachet-roles-bhuild#468 ·
I maintainer di solito rispondono entro 1 giorno