[BOT ISSUE] Google GenAI thinking model spans missing `thoughtsTokenCount` metric

Open Beginner friendly
#69 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
78/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Quiet
Tech stack
java
Domain
api, backend

Research direction

Start in braintrust-sdk/instrumentation/genai_1_18_0/src/main/java/com/google/genai/BraintrustApiClient.java, where tagSpan extracts usageMetadata, then review the established pattern in InstrumentationSemConv.java. Add coverage in BraintrustGenAITest.java for the thinking-model metric and verify the resulting completion_reasoning_tokens value.

Written by the indexing model from the issue text.

Description

Summary

The Google GenAI instrumentation extracts promptTokenCount, candidatesTokenCount, totalTokenCount, and cachedContentTokenCount from the usageMetadata object, but does not extract thoughtsTokenCount. This field is returned by Gemini thinking models (Gemini 2.5 Flash, Gemini 2.5 Pro, etc.) and represents the number of tokens used for internal reasoning — analogous to OpenAI's reasoning_tokens which this repo already captures for the Responses API.

Braintrust's own Gemini integration docs state that "reasoning tokens are automatically tracked" and "The wrapper captures completion_reasoning_tokens in the metrics" — but the Java SDK does not implement this.

What is missing

In BraintrustApiClient.tagSpan() (lines 128–149), the usageMetadata extraction handles four fields:

if (usage.containsKey("promptTokenCount")) {
    metrics.put("prompt_tokens", (Number) usage.get("promptTokenCount"));
}
if (usage.containsKey("candidatesTokenCount")) {
    metrics.put("completion_tokens", (Number) usage.get("candidatesTokenCount"));
}
if (usage.containsKey("totalTokenCount")) {
    metrics.put("tokens", (Number) usage.get("totalTokenCount"));
}
if (usage.containsKey("cachedContentTokenCount")) {
    metrics.put("prompt_cached_tokens", (Number) usage.get("cachedContentTokenCount"));
}

The missing extraction would be:

if (usage.containsKey("thoughtsTokenCount")) {
    metrics.put("completion_reasoning_tokens", (Number) usage.get("thoughtsTokenCount"));
}

This would align with:

  • The OpenAI Responses API handler in InstrumentationSemConv.tagOpenAIResponse() which already extracts output_tokens_details.reasoning_tokenscompletion_reasoning_tokens
  • The Braintrust docs which state completion_reasoning_tokens is captured for Gemini thinking models

Braintrust docs status

Upstream sources

  • Google GenAI thinking docs: https://ai.google.dev/gemini-api/docs/thinking — documents thoughtsTokenCount in usageMetadata as a standard field for thinking models, accessible via response.usage_metadata.thoughts_token_count
  • Google GenAI API reference: https://ai.google.dev/api/generate-contentUsageMetadata includes thoughtsTokenCount field
  • Gemini thinking models: Gemini 2.5 Flash and Gemini 2.5 Pro return this field by default when thinking is enabled

Local files inspected

  • braintrust-sdk/instrumentation/genai_1_18_0/src/main/java/com/google/genai/BraintrustApiClient.java — lines 128–149 (tagSpan usageMetadata extraction; thoughtsTokenCount not handled)
  • braintrust-sdk/src/main/java/dev/braintrust/instrumentation/InstrumentationSemConv.java — lines 144–150 (OpenAI output_tokens_details.reasoning_tokenscompletion_reasoning_tokens is the established pattern)
  • braintrust-sdk/instrumentation/genai_1_18_0/src/test/java/dev/braintrust/instrumentation/genai/v1_18_0/BraintrustGenAITest.java — no test uses a thinking model or asserts completion_reasoning_tokens
Dominant language
Java
Stars
21
Forks
5
Avg merge
2d 7h
Merged PRs (30d)
8

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from braintrustdata/braintrust-sdk-java

All issues in braintrustdata/braintrust-sdk-java

Similar issues

More Java issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.