[bot] Google GenAI Batches API is not instrumented (mis-tagged as a generic LLM span)

Bắt đầu với braintrust-sdk/instrumentation/genai_1_18_0/src/main/java/com/google/genai/BraintrustApiClient.java, đặc biệt là tagSpan(), sau đó kiểm tra BraintrustInstrumentation.java để xác nhận client và các entry point batch bất đồng bộ. So sánh các dạng của google-genai Batches API với instrumentation hiện có và xác định các test cần thêm cho create, get, list, cancel và các kết quả đã hoàn tất. Được xem là hoàn thành khi các thao tác batch có dữ liệu span phù hợp và độ bao phủ hồi quy, mà không coi metadata của BatchJob là đầu ra generation.

Summary

The genai_1_18_0 module instruments Google's google-genai Java SDK by subclassing its package-private ApiClient (com.google.genai.BraintrustApiClient) and swapping it into every service object on the Client, including client.batches and client.async.batches (BraintrustInstrumentation.wrapClient, lines 36–52). This means a call to client.batches.create(...)/.get(...)/.list(...)/.cancel(...)/.createEmbeddings(...) does produce a span — but BraintrustApiClient.tagSpan() has no awareness of the Batches API's request/response shape, so the span is mis-tagged: it's marked span_attributes.type = "llm" as if it were a real generation call, but the request/response field extraction (built for generateContent-shaped bodies) finds essentially nothing meaningful to populate.

What is missing

In braintrust-sdk/instrumentation/genai_1_18_0/src/main/java/com/google/genai/BraintrustApiClient.java, tagSpan() (lines 50–161):

• Request metadata extraction (lines 65–76) looks for top-level model, systemInstruction, tools, toolConfig, safetySettings, cachedContent. A batchGenerateContent request body wraps everything under a batch object ({"batch": {"displayName": ..., "inputConfig": {...}}}), and a createEmbeddings batch job body is shaped around EmbeddingsBatchJobSource/CreateEmbeddingsBatchJobConfig — none of the expected top-level fields exist, so metadata stays essentially empty.
• input_json construction (lines 97–113) only populates model/contents/config (from generationConfig) — a batch-create body has none of these at top level (the model is only present in the URL path, e.g. {model}:batchGenerateContent, and getModel(genAIEndpoint) (used as a fallback at line 102) is the only path by which model would end up in input_json at all); the batch's actual per-item contents/generation requests (whether inline or file/GCS-referenced) are never captured.
• Response handling (lines 116–150) dumps the whole response body as output_json (line 125) — for a batch create/get call, that whole body is just BatchJob resource metadata (name, state, createTime, etc.), not generative output — and reads usageMetadata for metrics (line 128), which a BatchJob response never has. There is no instrumentation at all of retrieving a completed batch's actual per-request results (where real generation outputs and usageMetadata become available), so even a fully successful batch job produces a span with type: "llm" but no meaningful input, output, or token metrics.
• No test or example anywhere in the repo exercises client.batches in any form (confirmed via grep for batch under braintrust-sdk/instrumentation/genai_1_18_0/ — the only match is BraintrustInstrumentation.java's own field-swap wiring, not a test).

Braintrust docs status: not_found

Checked https://www.braintrust.dev/docs/integrations/ai-providers/gemini in full: no mention of batchGenerateContent, Batches, or batch/async generation jobs anywhere, in any language. The only "batch"-adjacent mention is batchEmbedContents, and only in the unrelated context of AI proxy/gateway passthrough support for embeddings ("The gateway also supports Gemini's native embedContent and batchEmbedContents endpoints") — not span/tracing coverage, and not the Batches job API this issue is about.

Note: this repo's own gap-audit history already treats "Batch API not instrumented, mis-tagged as a generic LLM span" as a valid, in-scope finding — see the already-filed and still-open #155 for Anthropic's Message Batches API, which this issue mirrors for the Google GenAI provider (a distinct upstream API/SDK/module, not a duplicate).

Upstream sources

• Gemini API "Batch Mode" (batchGenerateContent) — official docs: https://ai.google.dev/gemini-api/docs/batch-api and https://ai.google.dev/gemini-api/docs/batch-mode. Async, 50%-discounted, ~24h turnaround; supports inline requests or file/JSONL upload.
• Vertex AI batch predictions for Gemini (and partner models) via GCS/BigQuery: https://docs.cloud.google.com/vertex-ai/generative-ai/docs/maas/capabilities/batch-prediction, with a Java-specific sample using the same com.google.genai.Batches class: https://docs.cloud.google.com/vertex-ai/generative-ai/docs/samples/googlegenaisdk-batchpredict-with-gcs.
• Official google-genai Java SDK exposes this directly: com.google.genai.Batches (googleapis/java-genai, src/main/java/com/google/genai/Batches.java), javadoc at https://googleapis.github.io/java-genai/javadoc/com/google/genai/Batches.html — create(String model, BatchJobSource, CreateBatchJobConfig), createEmbeddings(...) (experimental), get(...), cancel(...), delete(...), list(...), all returning/operating on BatchJob.

Local repo files inspected

• braintrust-sdk/instrumentation/genai_1_18_0/src/main/java/com/google/genai/BraintrustApiClient.java — lines 50–161 (tagSpan)
• braintrust-sdk/instrumentation/genai_1_18_0/src/main/java/com/google/genai/BraintrustInstrumentation.java — lines 22–57 (wrapClient), confirming client.batches/client.async.batches are explicitly wired to the same instrumented ApiClient (lines 37, 47) and therefore do produce (mis-tagged) spans rather than bypassing instrumentation entirely
• Repo-wide grep for batch/Batch/batchGenerateContent/BatchJob under braintrust-sdk/instrumentation/genai_1_18_0/ — only match is the field-swap wiring above; no test or example exercises the Batches API