Record Anthropic thinking tokens from the raw response (litellm reports 0)
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 52/100
Research direction
Start by locating the per-row usage recording and the code that writes reasoning tokens to usage_summary.csv, then inspect how the raw response usage is represented. Verify the change against the affected analysis CSVs, especially the tool_choice-auto sensitivity run, and confirm that provider responses with thinking report the expected reasoning count while other providers retain their existing fallback.
Written by the indexing model from the issue text.
Description
litellm 1.88.1 estimates Anthropic reasoning tokens by token-counting the visible thinking text instead of reading the API's usage.output_tokens_details.thinking_tokens; Claude Opus 5 / Fable 5 / Sonnet 5 default to display: "omitted" (empty thinking text), so completion_tokens_details.reasoning_tokens is structurally 0 for them. Filed upstream with a proposed fix: BerriAI/litellm#36290.
Until that lands, our per-row usage recording (and the published usage_summary.csv) shows zero reasoning tokens for every Claude row regardless of actual thinking. Options:
- Prefer the raw provider usage when available: read
output_tokens_details.thinking_tokensoff the response and record it in our reasoning column directly, falling back to litellm's field for other providers. - Regenerate the affected analysis CSVs for runs where thinking actually occurred (the tool_choice-auto sensitivity run is the first).
The board's canonical runs are unaffected in substance (Claude genuinely produced no thinking under the forced tool — see sensitivity/claude-thinking-2026-08.md), but the column should stop being wrong-by-construction before any v2 run where Claude thinking engages (#139).
🤖 Generated with Claude Code
- Dominant language
- Python
- Stars
- 2
- Forks
- 3
- Avg merge
- 11h 11m
- Merged PRs (30d)
- 22
Getting set up
- No Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from PolicyEngine/policybench
-
v2 prompt and data: program-specific disability facts, integer inputs, and the take-up boundaryOpen
Difficulty 5/5 Over a week Newbie friendliness 38/100
PolicyEngine/policybench#165 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
PolicyEngine/policybench#145 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 5/5 Over a week Newbie friendliness 25/100
PolicyEngine/policybench#139 · 2 comments ·
Maintainers usually reply within 1 day
-
Difficulty 3/5 1-2 days Newbie friendliness 68/100
PolicyEngine/policybench#129 ·
Maintainers usually reply within 1 day
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
PolicyEngine/policybench#118 ·
Maintainers usually reply within 1 day
All issues in PolicyEngine/policybench
Similar issues
-
Difficulty 1/5 1-3 hours Newbie friendliness 82/100
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 85/100
tharje/spooler-v2#20 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
Maintainers usually reply within 1 day
-
Bump version of azure-servicebusPossibly taken @joostlek claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
home-assistant/core#186275 ·
Maintainers usually reply within 1 day
-
documentation
Difficulty 1/5 Under an hour Newbie friendliness 72/100