[bot] OpenAI and Anthropic generation parameters not captured in span metadata
Nessuno ha ancora preso questa issue.
Valutazione
- Difficoltà
- 4/5
- Tempo stimato
- 3-5 giorni
- Idoneità per principianti
- 55/100
- Tipo di issue
- Funzionalità
- Chiarezza
- Abbastanza chiara
- Stato di attività
- Tranquilla
- Stack tecnologico
- java
- Ambito
- observability
Direzione di ricerca
Inizia in braintrust-sdk/src/main/java/dev/braintrust/instrumentation/InstrumentationSemConv.java, in tagOpenAIRequest e tagAnthropicRequest, quindi confronta l’estrazione dei parametri nel gestore Google GenAI di BraintrustApiClient.java. Esamina i file di test OpenAI e Anthropic esistenti e aggiungi la copertura per i parametri di generazione e le configurazioni specifiche del provider elencati. Il lavoro è completato quando i metadati della request supportati vengono acquisiti in modo coerente e i test lo verificano.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Descrizione
Summary
The OpenAI and Anthropic request taggers in InstrumentationSemConv only extract model and request routing info (request_path, request_base_uri, request_method) into span metadata. Generation parameters like temperature, max_tokens, tools, response_format, and provider-specific config like Anthropic's thinking are silently dropped.
In contrast, the Google GenAI handler in the same repo (BraintrustApiClient.tagSpan()) extracts temperature, topP, topK, maxOutputTokens, tools, toolConfig, safetySettings, responseMimeType, responseSchema, and more into metadata. This is an inconsistency within the repo — the Google GenAI handler provides materially more instrumentation detail for the same class of information.
What is missing
OpenAI — tagOpenAIRequest() (lines 78–108)
Currently captures only model in metadata. The following request parameters are silently dropped:
| Field | Purpose |
|---|---|
temperature |
Sampling temperature |
max_tokens / max_completion_tokens |
Output length limit |
top_p |
Nucleus sampling |
frequency_penalty, presence_penalty |
Repetition control |
tools |
Tool/function definitions |
response_format |
Structured output config (JSON mode, JSON Schema) |
reasoning_effort |
Reasoning effort for o-series models |
logprobs, top_logprobs |
Log probability settings |
stop |
Stop sequences |
For the Responses API, additional fields are missing: instructions, tools (with web_search, file_search, code_interpreter configs), reasoning (with effort and summary).
Anthropic — tagAnthropicRequest() (lines 163–205)
Currently captures only model in metadata. The following request parameters are silently dropped:
| Field | Purpose |
|---|---|
max_tokens |
Output length limit (required parameter) |
temperature |
Sampling temperature |
top_p, top_k |
Sampling parameters |
tools |
Tool definitions |
thinking |
Extended thinking config (type, budget_tokens) |
stop_sequences |
Stop sequences |
metadata |
User-provided metadata (e.g., user_id) |
The thinking config is particularly important: when a user enables extended thinking with a specific budget, this information is lost from the span. The Python SDK equivalent (issue braintrustdata/braintrust-sdk-python#107, now closed) captures thinking metadata.
Google GenAI — already captures these (for comparison)
BraintrustApiClient.tagSpan() (lines 64–95) extracts into metadata:
systemInstruction,tools,toolConfig,safetySettings,cachedContent- From
generationConfig:temperature,topP,topK,candidateCount,maxOutputTokens,stopSequences,responseMimeType,responseSchema
Impact
Users viewing traces in Braintrust can see generation parameters for Google GenAI calls but not for OpenAI or Anthropic calls. This makes it impossible to understand model configuration from the trace alone for the two most commonly used providers.
Braintrust docs status
- Braintrust tracing docs at https://www.braintrust.dev/docs/guides/tracing state that LLM spans show "the model, messages, parameters, token usage, and cost" — supported (parameters are documented as captured, but not implemented for OpenAI/Anthropic in this Java SDK)
- The Braintrust OpenAI docs mention temperature handling for GPT-5 models, implying it is a tracked parameter: unclear
Upstream sources
- OpenAI Chat Completions API: https://platform.openai.com/docs/api-reference/chat/create — documents all request parameters including
temperature,max_tokens,tools,response_format,reasoning_effort - OpenAI Responses API: https://platform.openai.com/docs/api-reference/responses/create — documents
instructions,tools,reasoning - Anthropic Messages API: https://docs.anthropic.com/en/api/messages — documents
max_tokens,temperature,tools,thinking,top_p,top_k,stop_sequences - Anthropic extended thinking: https://docs.anthropic.com/en/docs/build-with-claude/extended-thinking — documents
thinkingparameter withtypeandbudget_tokens
Local files inspected
braintrust-sdk/src/main/java/dev/braintrust/instrumentation/InstrumentationSemConv.java— lines 78–108 (tagOpenAIRequest: onlymodelextracted), lines 163–205 (tagAnthropicRequest: onlymodelextracted)braintrust-sdk/instrumentation/genai_1_18_0/src/main/java/com/google/genai/BraintrustApiClient.java— lines 64–95 (tagSpan: comprehensive parameter extraction into metadata)braintrust-sdk/instrumentation/openai_2_8_0/src/test/java/dev/braintrust/instrumentation/openai/v2_8_0/BraintrustOpenAITest.java— no test asserts generation parameters in metadatabraintrust-sdk/instrumentation/anthropic_2_2_0/src/test/java/dev/braintrust/instrumentation/anthropic/v2_2_0/BraintrustAnthropicTest.java— no test asserts generation parameters in metadata
- Lingua principale
- Java
- Stelle
- 21
- Fork
- 5
- Merge medio
- 2g 7h
- PR unite (30g)
- 8
Guida per i contributori
Apri la guida per i contributori
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Altre issue di braintrustdata/braintrust-sdk-java
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 82/100
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 82/100
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 82/100
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 78/100
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 68/100
Tutte le issue di braintrustdata/braintrust-sdk-java
Issue simili
-
awaiting triage bug Causes friction Hop Gui P1 P2 Transforms
Difficoltà 2/5 1-3 ore Idoneità per principianti 75/100
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 75/100
apache/flink-agents#1152 ·
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 75/100
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 70/100
jenkinsci/blueocean-plugin#5417 ·
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 75/100
objectionary/eo-graphs#75 ·