[BOT ISSUE] OpenAI embeddings spans missing output capture and incomplete input extraction

Open Beginner friendly
#66 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
68/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Quiet
Tech stack
java

Research direction

Start in braintrust-sdk/src/main/java/dev/braintrust/instrumentation/InstrumentationSemConv.java, reviewing tagOpenAIRequest(), tagOpenAIResponse(), and the Embeddings span mapping near line 244. Check BraintrustOpenAITest.java and the OpenAI embeddings API shapes; done means single-string inputs and data[].embedding responses populate span data without regressing Chat Completions or Responses handling.

Written by the indexing model from the issue text.

Description

Summary

The OpenAI instrumentation recognizes embeddings calls (span is named "Embeddings") but the span data is largely empty because the response and request parsing logic in InstrumentationSemConv only handles Chat Completions and Responses API field shapes.

Specifically:

  • No output_json: the response parser checks for choices (Chat Completions) or output (Responses API), but embeddings responses have data[].embedding — neither branch matches
  • Incomplete input_json: the request parser checks for messages or input (as array), but embeddings input can be a single string, which fails the isArray() check on line 102
  • Partial metrics: prompt_tokens and total_tokens are captured correctly from usage, but completion_tokens is absent (embeddings don't produce completion tokens) — this is fine but worth noting

What is missing

In InstrumentationSemConv.tagOpenAIRequest() (lines 78–108):

if (requestJson.has("messages")) {
    span.setAttribute("braintrust.input_json", toJson(requestJson.get("messages")));
} else if (requestJson.has("input") && requestJson.get("input").isArray()) {
    span.setAttribute("braintrust.input_json", toJson(requestJson.get("input")));
}
  • Embeddings requests use input which can be a string ("Hello world"), an array of strings, or an array of token arrays. Single-string inputs fail the isArray() guard and are silently dropped.

In InstrumentationSemConv.tagOpenAIResponse() (lines 111–156):

if (responseJson.has("choices")) {
    span.setAttribute("braintrust.output_json", toJson(responseJson.get("choices")));
} else if (responseJson.has("output")) {
    span.setAttribute("braintrust.output_json", toJson(responseJson.get("output")));
}
  • Embeddings responses have data (array of embedding objects) and model — neither choices nor output. No output_json is set.

Additionally, the model field from the embeddings request (e.g., text-embedding-3-small) is correctly extracted into metadata, and usage metrics partially work — so this is a detail gap rather than total failure.

Braintrust docs status

  • The Braintrust OpenAI integration docs at braintrust.dev/docs/integrations/ai-providers/openai do not mention embeddings: not_found
  • No embeddings instrumentation is documented for any provider

Upstream sources

Local files inspected

  • braintrust-sdk/src/main/java/dev/braintrust/instrumentation/InstrumentationSemConv.java — lines 78–108 (tagOpenAIRequest: input string case not handled), lines 111–156 (tagOpenAIResponse: no data field extraction), line 244 (span name correctly maps to "Embeddings")
  • braintrust-sdk/instrumentation/openai_2_8_0/src/main/java/dev/braintrust/instrumentation/openai/v2_8_0/TracingHttpClient.java — HTTP-level wrapping captures the call, delegates to InstrumentationSemConv
  • braintrust-sdk/instrumentation/openai_2_8_0/src/test/java/dev/braintrust/instrumentation/openai/v2_8_0/BraintrustOpenAITest.java — no embeddings test exists
Dominant language
Java
Stars
21
Forks
5
Avg merge
2d 7h
Merged PRs (30d)
8

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from braintrustdata/braintrust-sdk-java

All issues in braintrustdata/braintrust-sdk-java

Similar issues

More Java issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.