[Bug]: Document whether `prompt_tokens` includes `cache_read_tokens` in accumulated token usage

Open Beginner friendly
#5,975 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

@Jaxon1216 is already working on this.

Since Oct 15, 2026.

  • #6001 by @Jaxon1216 — open

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
72/100
Issue type
Documentation
Clarity
Clearly specified
Activity status
Active
Tech stack
python

Research direction

Start in openhands-sdk/openhands/sdk/llm/utils/metrics.py: read the TokenUsage field definitions for prompt_tokens and cache_read_tokens, then MetricsSnapshot.cache_hit_rate, which already encodes the denominator rule. Read normalize_usage in telemetry.py to confirm the OpenAI-style mapping. Done when the field descriptions state the overlap rule consistently with cache_hit_rate, and tests/sdk/llm/test_llm_metrics.py still passes unchanged.

Written by the indexing model from the issue text.

Description

priority:low ready-for-dev
Is there an existing issue for the same bug?
  • I have searched existing issues and this is not a duplicate.
Bug Description

The SDK exposes accumulated token usage with both prompt_tokens and cache_read_tokens, but the public field-level documentation does not state whether cached tokens are included inside prompt_tokens or reported as a disjoint bucket.

This matters for anyone computing input-token totals or cache-hit rates from accumulated_token_usage: adding prompt_tokens + cache_read_tokens double-counts for providers whose prompt_tokens already includes cached reads, while treating them as fully disjoint undercounts for providers that report cached reads separately.

From code inspection, the intended semantics appear to be provider-shape-dependent:

  • For LiteLLM/OpenAI Chat Completions usage, normalize_usage copies usage.prompt_tokens to prompt_tokens and prompt_tokens_details.cached_tokens to cache_read_tokens; cached reads are therefore a detail/subset of the prompt total.
  • The cache_hit_rate implementation already encodes this interpretation: it normally uses prompt_tokens as the denominator and only switches to prompt_tokens + cache_read_tokens when cache_read_tokens > prompt_tokens (the separately-reported/ACP-style case).

That invariant is currently only discoverable by reading the metrics/telemetry implementation. It should be explicit in the model field documentation and/or SDK metrics docs so API consumers do not have to infer it.

Steps to Reproduce
  1. Inspect the token-usage model fields for accumulated_token_usage (prompt_tokens, cache_read_tokens).
  2. Try to determine from the public field descriptions whether cache_read_tokens is included in prompt_tokens.
  3. Observe that the field descriptions only say the values must be non-negative; the overlap rule is not stated there.
Actual Behavior

Public consumers cannot tell whether to compute total input tokens as prompt_tokens or as prompt_tokens + cache_read_tokens without reading SDK internals.

Expected Behavior

The overlap semantics are documented at the point of use:

  • For LiteLLM/OpenAI-style usage: cache_read_tokens is included in prompt_tokens.
  • For providers/ACP-style reporting where cached reads are reported separately: consumers should follow the same denominator rule used by cache_hit_rate rather than assuming the fields are always disjoint or always overlapping.
Relevant Code
  • openhands-sdk/openhands/sdk/llm/utils/telemetry.py: normalize_usage maps provider usage into prompt_tokens and cache_read_tokens.
  • openhands-sdk/openhands/sdk/llm/utils/metrics.py: TokenUsage fields and MetricsSnapshot.cache_hit_rate denominator logic.
  • TokenUsage.__add__ sums the fields independently, which is consistent with prompt_tokens being the prompt total and cache_read_tokens being a cached-read detail for OpenAI-style usage.
Acceptance Criteria
  • TokenUsage.prompt_tokens and TokenUsage.cache_read_tokens field descriptions (or the SDK metrics documentation) explicitly state when cached reads are included in prompt_tokens and when they are reported separately.
  • The documented total-input-token guidance matches the denominator rule already implemented by MetricsSnapshot.cache_hit_rate.
  • Existing normalization/accumulation behavior is covered by a test or doc example showing an OpenAI-style usage object with cached tokens and verifying that consumers should not double-count them.
  • No behavior change is required unless maintainers decide the current provider-dependent semantics should be normalized instead; in that case the migration/behavior change is called out explicitly in the fix.
Additional Context

This came from a documentation/API-clarity review of the SDK metrics surface. No private user data, account information, or conversation data is included.

This issue was filed by an AI agent (OpenHands) on behalf of the user.


OpenHands AI triage

The following comments and acceptance criteria were added by the OpenHands AI agent.

Triage

This is an in-scope documentation-clarity issue owned by this repository. TokenUsage lives in openhands-sdk/openhands/sdk/llm/utils/metrics.py, and normalize_usage in openhands-sdk/openhands/sdk/llm/utils/telemetry.py maps LiteLLM/OpenAI prompt_tokens plus prompt_tokens_details.cached_tokens into prompt_tokens and cache_read_tokens. The intended rule is already encoded in MetricsSnapshot.cache_hit_rate: cached reads are a subset of prompt_tokens for OpenAI-style usage, and are reported separately for provider/ACP-style usage where cache_read_tokens can exceed prompt_tokens.

Smallest coherent scope: state the overlap rule in the public TokenUsage field descriptions and align the wording with the existing cache_hit_rate denominator rule, so a consumer can compute input totals without reading internals. No in-repo metrics markdown guide exists (SDK docs live at docs.openhands.dev), so the field descriptions are the authoritative in-repo surface.

Non-goals: no change to normalize_usage, TokenUsage.__add__, merge, diff, or cache_hit_rate behavior; no new token fields; no provider-specific normalization; no change to cache_write_tokens semantics.

Acceptance Criteria
  • TokenUsage.prompt_tokens and TokenUsage.cache_read_tokens field descriptions state that, for LiteLLM/OpenAI-style usage, cache_read_tokens is included in (a subset of) prompt_tokens, and that for provider/ACP-style usage cached reads are reported separately, so cache_read_tokens may exceed prompt_tokens.
  • The documented guidance for total input tokens matches the MetricsSnapshot.cache_hit_rate denominator rule: use prompt_tokens, switching to prompt_tokens + cache_read_tokens only when cache_read_tokens > prompt_tokens.
  • The cache_hit_rate docstring and the new field descriptions use consistent terminology (included/subset versus reported separately) so the two surfaces do not contradict each other.
  • A documentation example (docstring or SDK docs) or a behavior test shows an OpenAI-style usage object with cached tokens and states that consumers must not add cache_read_tokens to prompt_tokens again; if written as a test it asserts the resulting total or cache_hit_rate, not docstring text.
  • Existing tests in tests/sdk/llm/test_llm_metrics.py pass unchanged, confirming no runtime behavior change.
Dominant language
Python
Stars
1.4k
Forks
797
Avg merge
1d 17h
Merged PRs (30d)
354

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from OpenHands/software-agent-sdk

All issues in OpenHands/software-agent-sdk

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.