Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Google GenAI `generateContentStream` aggregation silently drops `inlineData` output parts (images, audio)

Closed Beginner friendly
  • Is something wrong?
#1,690 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
68/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Quiet
Tech stack
typescript
Domain
observability

Research direction

Start in js/src/instrumentation/plugins/google-genai-plugin.ts at aggregateGenerateContentChunks and compare its part handling with the inlineData declaration in js/src/vendor-sdk-types/google-genai.ts. Review e2e/scenarios/google-genai-instrumentation/ for the existing instrumentation checks. Done means streamed inlineData parts are retained in the aggregated output, with coverage for image or other binary output.

Written by the indexing model from the issue text.

Description

stale

Summary

The Google GenAI streaming aggregation code drops inlineData parts from the response output. When Gemini models generate images natively via generateContent with responseModalities: ['IMAGE'] (or return audio/other binary data), the inlineData parts in streamed chunks are silently lost because the aggregation loop does not handle them.

Non-streaming generateContent calls are unaffected — the plugin logs the full raw response as output.

What instrumentation is missing

In js/src/instrumentation/plugins/google-genai-plugin.ts, the aggregateGenerateContentChunks function (lines 749–778) processes parts from streamed chunks:

for (const part of candidate.content.parts) {
  if (part.text !== undefined) {
    // handled ✓
  } else if (part.functionCall) {
    // handled ✓
  } else if (part.codeExecutionResult) {
    // handled ✓
  } else if (part.executableCode) {
    // handled ✓
  }
  // inlineData → falls through, silently dropped ✗
}

The vendored type GoogleGenAIPart in js/src/vendor-sdk-types/google-genai.ts already declares the inlineData field (line 53), but the aggregation code never handles it. Any inlineData part in a streamed chunk is silently excluded from the aggregated output span.

Impact

  • Native image generation via Gemini models (gemini-2.0-flash, etc.) with streaming produces spans where generated images are missing from the output
  • Braintrust docs state "Streaming responses are fully supported — Braintrust automatically collects streamed chunks and logs the complete response as a single span," but this is not the case for image/audio output
  • Users who stream generateContent calls with responseModalities: ['IMAGE', 'TEXT'] will see text in their spans but not the generated images

Braintrust docs status

unclear — Braintrust docs at https://www.braintrust.dev/docs/instrument/wrap-providers list @google/genai as supported and claim full streaming support, but do not specifically address image output in streamed responses.

Upstream reference

  • Google GenAI native image generation: https://ai.google.dev/gemini-api/docs/image-generation
  • generateContent with responseModalities: ['IMAGE'] returns inlineData parts containing generated images
  • This is a stable feature available on Gemini 2.0 Flash and later models

Local files inspected

  • js/src/instrumentation/plugins/google-genai-plugin.ts (lines 749–778: aggregateGenerateContentChunks part processing loop)
  • js/src/vendor-sdk-types/google-genai.ts (line 53: inlineData field on GoogleGenAIPart)
  • js/src/wrappers/google-genai.ts (wrapper proxies generateContentStream to channel)
  • e2e/scenarios/google-genai-instrumentation/ (no test cases with image output in streamed responses)

Note

This is distinct from #1673 (models.generateImages() not instrumented), which covers the dedicated Imagen API. This issue is about the standard generateContent/generateContentStream API producing image output that gets lost specifically in the streaming aggregation path.

Dominant language
TypeScript
Stars
28
Forks
14
Avg merge
1d 23h
Merged PRs (30d)
68

Getting set up

  • Ships a Dockerfile or Docker Compose file
  • No pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from braintrustdata/braintrust-sdk-javascript

All issues in braintrustdata/braintrust-sdk-javascript

Similar issues

More TypeScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.