feat(run): report cost and stop a run at a budget
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 30/100
Research direction
Start by searching internal/ for token fields in the result types, then read the retry loop in internal/evaluator/evaluator.go around line 362, since retried attempts must be counted too. Agree the design first: where prices come from in the user config, what counts as partial, and how the ceiling interacts with #350's planned count. Done means per-run, per-case and per-suite cost appear in results, and --max-cost stops new runs and exits non-zero.
Written by the indexing model from the issue text.
Description
Problem / motivation
It is hard to see what a suite will cost, and nothing stops a run that goes over budget.
--dry-runreports cases, not agent runs. With the code-stats example,skill-up run evals/eval.yaml --dry-run --baseline --iteration 5printsWould run 4 case(s). That is 40 agent runs (4 cases, 2 arms, 5 iterations), before any agent-judge or user-simulator calls.- Results record tokens, not money. I found no cost or price fields in
internal/. - Case timeouts and turn limits bound each case, but nothing caps what a whole suite spends.
Cost is a stated reason not to adopt. One team's assessment concluded "评测本身消耗模型调用费用,ROI 在当前规模下为负" (the evals' own model calls make the ROI negative at our scale) (assessment). It is also the first question before running evals in CI on every pull request.
Proposed solution
- Cost in results. Report it per run, per case and per suite, split into agent, agent judge and user simulator. Use the engine's own figure when it reports one (Claude Code's JSON result includes
total_cost_usd). Otherwise multiply tokens by a per-model price table in the user config, since prices change and differ by provider. Without a price, show tokens only, as today. Count every attempt, including retried ones: a retry replaces the earlier attempt's result (evaluator.go#L362), so pricing only the final results would undercount. - A budget ceiling:
--max-cost <amount>and aneval.yamlfield. Once the spent cost passes the limit, stop starting new runs and let running ones finish. Mark the result partial with the reason, and exit non-zero. - A token ceiling (
--max-tokens) for users without a price table.
Alternatives considered
- Provider dashboards. They show the bill after the fact, not per suite, and not in time to stop a run.
- A token ceiling only. That works without prices, but the same token count costs very different amounts across models.
Additional context
- #350 plans to report the planned case and iteration count before paid execution. This issue covers cost and the ceiling, and would build on that count.
compare --max-token-increase-percentalready gates token growth between two results.- Anthropic's plugin evals show a cost column and take
--max-cost-usd, which marks a run partial when hit (docs).
- Dominant language
- Go
- Stars
- 1.5k
- Forks
- 134
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 61
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from alibaba/skill-up
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
Maintainers usually reply within 1 day
-
OpenSandbox runtime: expect.files_exist and file_contains fail because WorkspacePath is remotePossibly taken @bababoom721 claimed this 1 day ago. Open
Difficulty 4/5 3-5 days Newbie friendliness 25/100
Maintainers usually reply within 1 day
-
Difficulty 4/5 3-5 days Newbie friendliness 30/100
alibaba/skill-up#408 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 5/5 Over a week Newbie friendliness 15/100
alibaba/skill-up#407 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 4/5 3-5 days Newbie friendliness 25/100
alibaba/skill-up#406 · 1 comment ·
Maintainers usually reply within 1 day
All issues in alibaba/skill-up
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
OneBusAway/maglev#2145 ·
Maintainers usually reply within 2 days
-
priority/medium status/ready type/chore
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
discobox-ai/discobox#1357 ·
Maintainers usually reply within 1 day
-
TestPseudoLocaleRemote fails now and then: it types the code before the laptop asks for itPossibly taken @fzn0x claimed this today. Openbug priority: low
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
Maintainers usually reply within 1 day
-
edit_file diff rendering panics on invalid UTF-8 textPossibly taken @TDnorthgarden claimed this today. Openarea/tui
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
docker/docker-agent#4955 ·
Maintainers usually reply within 1 day
-
enhancement ui/ux
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
jbigler/railswitch#243 ·
Maintainers usually reply within 1 day