feat(run): report cost and stop a run at a budget

Aperta
#405 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
30/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
go
Ambito
cli, data

Direzione di ricerca

Start by searching internal/ for token fields in the result types, then read the retry loop in internal/evaluator/evaluator.go around line 362, since retried attempts must be counted too. Agree the design first: where prices come from in the user config, what counts as partial, and how the ceiling interacts with #350's planned count. Done means per-run, per-case and per-suite cost appear in results, and --max-cost stops new runs and exits non-zero.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Problem / motivation

It is hard to see what a suite will cost, and nothing stops a run that goes over budget.

  • --dry-run reports cases, not agent runs. With the code-stats example, skill-up run evals/eval.yaml --dry-run --baseline --iteration 5 prints Would run 4 case(s). That is 40 agent runs (4 cases, 2 arms, 5 iterations), before any agent-judge or user-simulator calls.
  • Results record tokens, not money. I found no cost or price fields in internal/.
  • Case timeouts and turn limits bound each case, but nothing caps what a whole suite spends.

Cost is a stated reason not to adopt. One team's assessment concluded "评测本身消耗模型调用费用,ROI 在当前规模下为负" (the evals' own model calls make the ROI negative at our scale) (assessment). It is also the first question before running evals in CI on every pull request.

Proposed solution

  1. Cost in results. Report it per run, per case and per suite, split into agent, agent judge and user simulator. Use the engine's own figure when it reports one (Claude Code's JSON result includes total_cost_usd). Otherwise multiply tokens by a per-model price table in the user config, since prices change and differ by provider. Without a price, show tokens only, as today. Count every attempt, including retried ones: a retry replaces the earlier attempt's result (evaluator.go#L362), so pricing only the final results would undercount.
  2. A budget ceiling: --max-cost <amount> and an eval.yaml field. Once the spent cost passes the limit, stop starting new runs and let running ones finish. Mark the result partial with the reason, and exit non-zero.
  3. A token ceiling (--max-tokens) for users without a price table.

Alternatives considered

  • Provider dashboards. They show the bill after the fact, not per suite, and not in time to stop a run.
  • A token ceiling only. That works without prices, but the same token count costs very different amounts across models.

Additional context

  • #350 plans to report the planned case and iteration count before paid execution. This issue covers cost and the ceiling, and would build on that count.
  • compare --max-token-increase-percent already gates token growth between two results.
  • Anthropic's plugin evals show a cost column and take --max-cost-usd, which marks a run partial when hit (docs).
Lingua principale
Go
Stelle
1.5k
Fork
134
Merge medio
1g 22h
PR unite (30g)
61

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di alibaba/skill-up

Tutte le issue di alibaba/skill-up

Issue simili

Altre issue su Go

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.