[BOT ISSUE] Google GenAI thinking model spans missing `thoughtsTokenCount` metric

Abierto Apto para principiantes
#69 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
2/5
Tiempo estimado
1-3 horas
Aptitud para principiantes
78/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Tranquilo
Stack tecnológico
java
Área
api, backend

Línea de trabajo

Comienza en braintrust-sdk/instrumentation/genai_1_18_0/src/main/java/com/google/genai/BraintrustApiClient.java, donde tagSpan extrae usageMetadata; después, revisa el patrón establecido en InstrumentationSemConv.java. Añade cobertura en BraintrustGenAITest.java para la métrica de los modelos de razonamiento y verifica el valor resultante de completion_reasoning_tokens.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Summary

The Google GenAI instrumentation extracts promptTokenCount, candidatesTokenCount, totalTokenCount, and cachedContentTokenCount from the usageMetadata object, but does not extract thoughtsTokenCount. This field is returned by Gemini thinking models (Gemini 2.5 Flash, Gemini 2.5 Pro, etc.) and represents the number of tokens used for internal reasoning — analogous to OpenAI's reasoning_tokens which this repo already captures for the Responses API.

Braintrust's own Gemini integration docs state that "reasoning tokens are automatically tracked" and "The wrapper captures completion_reasoning_tokens in the metrics" — but the Java SDK does not implement this.

What is missing

In BraintrustApiClient.tagSpan() (lines 128–149), the usageMetadata extraction handles four fields:

if (usage.containsKey("promptTokenCount")) {
    metrics.put("prompt_tokens", (Number) usage.get("promptTokenCount"));
}
if (usage.containsKey("candidatesTokenCount")) {
    metrics.put("completion_tokens", (Number) usage.get("candidatesTokenCount"));
}
if (usage.containsKey("totalTokenCount")) {
    metrics.put("tokens", (Number) usage.get("totalTokenCount"));
}
if (usage.containsKey("cachedContentTokenCount")) {
    metrics.put("prompt_cached_tokens", (Number) usage.get("cachedContentTokenCount"));
}

The missing extraction would be:

if (usage.containsKey("thoughtsTokenCount")) {
    metrics.put("completion_reasoning_tokens", (Number) usage.get("thoughtsTokenCount"));
}

This would align with:

  • The OpenAI Responses API handler in InstrumentationSemConv.tagOpenAIResponse() which already extracts output_tokens_details.reasoning_tokenscompletion_reasoning_tokens
  • The Braintrust docs which state completion_reasoning_tokens is captured for Gemini thinking models

Braintrust docs status

Upstream sources

  • Google GenAI thinking docs: https://ai.google.dev/gemini-api/docs/thinking — documents thoughtsTokenCount in usageMetadata as a standard field for thinking models, accessible via response.usage_metadata.thoughts_token_count
  • Google GenAI API reference: https://ai.google.dev/api/generate-contentUsageMetadata includes thoughtsTokenCount field
  • Gemini thinking models: Gemini 2.5 Flash and Gemini 2.5 Pro return this field by default when thinking is enabled

Local files inspected

  • braintrust-sdk/instrumentation/genai_1_18_0/src/main/java/com/google/genai/BraintrustApiClient.java — lines 128–149 (tagSpan usageMetadata extraction; thoughtsTokenCount not handled)
  • braintrust-sdk/src/main/java/dev/braintrust/instrumentation/InstrumentationSemConv.java — lines 144–150 (OpenAI output_tokens_details.reasoning_tokenscompletion_reasoning_tokens is the established pattern)
  • braintrust-sdk/instrumentation/genai_1_18_0/src/test/java/dev/braintrust/instrumentation/genai/v1_18_0/BraintrustGenAITest.java — no test uses a thinking model or asserts completion_reasoning_tokens
Lenguaje dominante
Java
Estrellas
21
Forks
5
Merge medio
2 d 7 h
PR fusionados (30 d)
8

Guía de contribución

Abrir la guía de contribución

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de braintrustdata/braintrust-sdk-java

Todos los issues de braintrustdata/braintrust-sdk-java

Issues similares

Más issues de Java

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.