feat(run): report cost and stop a run at a budget

Open
#405 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
30/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
go
Domain
cli, data

Research direction

Start by searching internal/ for token fields in the result types, then read the retry loop in internal/evaluator/evaluator.go around line 362, since retried attempts must be counted too. Agree the design first: where prices come from in the user config, what counts as partial, and how the ceiling interacts with #350's planned count. Done means per-run, per-case and per-suite cost appear in results, and --max-cost stops new runs and exits non-zero.

Written by the indexing model from the issue text.

Description

Problem / motivation

It is hard to see what a suite will cost, and nothing stops a run that goes over budget.

  • --dry-run reports cases, not agent runs. With the code-stats example, skill-up run evals/eval.yaml --dry-run --baseline --iteration 5 prints Would run 4 case(s). That is 40 agent runs (4 cases, 2 arms, 5 iterations), before any agent-judge or user-simulator calls.
  • Results record tokens, not money. I found no cost or price fields in internal/.
  • Case timeouts and turn limits bound each case, but nothing caps what a whole suite spends.

Cost is a stated reason not to adopt. One team's assessment concluded "评测本身消耗模型调用费用,ROI 在当前规模下为负" (the evals' own model calls make the ROI negative at our scale) (assessment). It is also the first question before running evals in CI on every pull request.

Proposed solution

  1. Cost in results. Report it per run, per case and per suite, split into agent, agent judge and user simulator. Use the engine's own figure when it reports one (Claude Code's JSON result includes total_cost_usd). Otherwise multiply tokens by a per-model price table in the user config, since prices change and differ by provider. Without a price, show tokens only, as today. Count every attempt, including retried ones: a retry replaces the earlier attempt's result (evaluator.go#L362), so pricing only the final results would undercount.
  2. A budget ceiling: --max-cost <amount> and an eval.yaml field. Once the spent cost passes the limit, stop starting new runs and let running ones finish. Mark the result partial with the reason, and exit non-zero.
  3. A token ceiling (--max-tokens) for users without a price table.

Alternatives considered

  • Provider dashboards. They show the bill after the fact, not per suite, and not in time to stop a run.
  • A token ceiling only. That works without prices, but the same token count costs very different amounts across models.

Additional context

  • #350 plans to report the planned case and iteration count before paid execution. This issue covers cost and the ceiling, and would build on that count.
  • compare --max-token-increase-percent already gates token growth between two results.
  • Anthropic's plugin evals show a cost column and take --max-cost-usd, which marks a run partial when hit (docs).
Dominant language
Go
Stars
1.5k
Forks
134
Avg merge
1d 22h
Merged PRs (30d)
61

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from alibaba/skill-up

All issues in alibaba/skill-up

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.