Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

Google GenAI `generateContentStream` aggregation silently drops `inlineData` output parts (images, audio)

Cerrado Apto para principiantes
  • ¿Algo no está bien?
#1,690 1 comentario 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
2/5
Tiempo estimado
1-3 horas
Aptitud para principiantes
68/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Tranquilo
Stack tecnológico
typescript
Área
observability

Línea de trabajo

Empieza en js/src/instrumentation/plugins/google-genai-plugin.ts, en aggregateGenerateContentChunks, y compara el tratamiento de part con la declaración de inlineData en js/src/vendor-sdk-types/google-genai.ts. Revisa e2e/scenarios/google-genai-instrumentation/ para comprobar las verificaciones de instrumentación existentes. Se considera terminado cuando las partes inlineData transmitidas se conservan en la salida agregada y existe cobertura para imágenes u otras salidas binarias.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

stale

Summary

The Google GenAI streaming aggregation code drops inlineData parts from the response output. When Gemini models generate images natively via generateContent with responseModalities: ['IMAGE'] (or return audio/other binary data), the inlineData parts in streamed chunks are silently lost because the aggregation loop does not handle them.

Non-streaming generateContent calls are unaffected — the plugin logs the full raw response as output.

What instrumentation is missing

In js/src/instrumentation/plugins/google-genai-plugin.ts, the aggregateGenerateContentChunks function (lines 749–778) processes parts from streamed chunks:

for (const part of candidate.content.parts) {
  if (part.text !== undefined) {
    // handled ✓
  } else if (part.functionCall) {
    // handled ✓
  } else if (part.codeExecutionResult) {
    // handled ✓
  } else if (part.executableCode) {
    // handled ✓
  }
  // inlineData → falls through, silently dropped ✗
}

The vendored type GoogleGenAIPart in js/src/vendor-sdk-types/google-genai.ts already declares the inlineData field (line 53), but the aggregation code never handles it. Any inlineData part in a streamed chunk is silently excluded from the aggregated output span.

Impact

  • Native image generation via Gemini models (gemini-2.0-flash, etc.) with streaming produces spans where generated images are missing from the output
  • Braintrust docs state "Streaming responses are fully supported — Braintrust automatically collects streamed chunks and logs the complete response as a single span," but this is not the case for image/audio output
  • Users who stream generateContent calls with responseModalities: ['IMAGE', 'TEXT'] will see text in their spans but not the generated images

Braintrust docs status

unclear — Braintrust docs at https://www.braintrust.dev/docs/instrument/wrap-providers list @google/genai as supported and claim full streaming support, but do not specifically address image output in streamed responses.

Upstream reference

  • Google GenAI native image generation: https://ai.google.dev/gemini-api/docs/image-generation
  • generateContent with responseModalities: ['IMAGE'] returns inlineData parts containing generated images
  • This is a stable feature available on Gemini 2.0 Flash and later models

Local files inspected

  • js/src/instrumentation/plugins/google-genai-plugin.ts (lines 749–778: aggregateGenerateContentChunks part processing loop)
  • js/src/vendor-sdk-types/google-genai.ts (line 53: inlineData field on GoogleGenAIPart)
  • js/src/wrappers/google-genai.ts (wrapper proxies generateContentStream to channel)
  • e2e/scenarios/google-genai-instrumentation/ (no test cases with image output in streamed responses)

Note

This is distinct from #1673 (models.generateImages() not instrumented), which covers the dedicated Imagen API. This issue is about the standard generateContent/generateContentStream API producing image output that gets lost specifically in the streaming aggregation path.

Lenguaje dominante
TypeScript
Estrellas
29
Forks
16
Merge medio
1 d 20 h
PR fusionados (30 d)
70

Preparar el entorno

  • Incluye un Dockerfile o un archivo de Docker Compose
  • Sin plantilla de pull request
  • Sin guía de contribución

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de braintrustdata/braintrust-sdk-javascript

Todos los issues de braintrustdata/braintrust-sdk-javascript

Issues similares

Más issues de TypeScript

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.