[bot] Bedrock Runtime plugin does not minimize output or capture token metrics for embedding models (Titan Embeddings, Cohere Embed)

Aberta
  • Algo errado?
#2,303 1 comentário 0 reações 0 responsáveis Ver no GitHub

Mantenedores costumam responder em até 1 dia

Ninguém assumiu esta issue ainda.

  • #2475 de @lforst — fechado sem integrar
  • #2661 de @Tyagiquamar — fechado sem integrar

Avaliação

Dificuldade
3/5
Tempo estimado
1-2 dias
Facilidade para iniciantes
68/100
Tipo de issue
Bug
Clareza
Razoavelmente clara
Status de atividade
Ativa
Stack de tecnologia
aws, typescript
Domínio
observability

Direção de pesquisa

Comece em js/src/instrumentation/plugins/bedrock-runtime-plugin.ts, em extractBedrockRuntimeOutput e parseBedrockRuntimeMetrics; depois, revise bedrock-runtime-common.ts e o tratamento de embeddings em openai-plugin.ts. Está concluído quando as respostas de embeddings do Bedrock forem resumidas pelo comprimento do embedding em vez de registrar os vetores, e o inputTextTokenCount do Titan for exposto como uso de tokens para chamadas invokeModel.

Escrita pelo modelo de indexação a partir do texto da issue.

Descrição

stale

Summary

The @aws-sdk/client-bedrock-runtime InvokeModelCommand surface is used both for text-generation models and for embeddings models (Amazon Titan Text Embeddings V2, Cohere Embed v3/v4 on Bedrock). This repo's Bedrock Runtime plugin treats every invokeModel call identically — it does not detect embeddings responses, so it logs the full raw embedding vector(s) as span output instead of a minimized summary, and it fails to extract token-usage metrics for Titan Embeddings because the plugin's usage-parsing logic only looks for a nested usage/metrics object, which Titan's response does not have.

This is a "materially less detail / less consistent" gap relative to every other embeddings-capable provider already instrumented in this repo: OpenAI, Cohere (direct), Mistral, Groq, and HuggingFace all special-case their embeddings endpoints to log { embedding_length: N } (or equivalent) instead of the raw vector, and to parse token usage from the embeddings response. Bedrock's invokeModel path has no such special-casing at all.

What instrumentation is missing

In js/src/instrumentation/plugins/bedrock-runtime-plugin.ts:

  • extractBedrockRuntimeOutput (line 151): for operation === "invokeModel" it does parseJsonBody(response?.body) ?? summarizeBody(response?.body) — fully generic, no branch for embedding responses.
  • extractBedrockRuntimeResponseMetrics (line 205) → parseBedrockRuntimeMetrics (line 229): looks for inputTokens / inputTokenCount / input_tokens / prompt_tokens inside a usage (or metadata.usage) object. Titan Embeddings V2's response is { "embedding": [...], "inputTextTokenCount": <int> } — there is no usage object at all, and the field is named inputTextTokenCount, which none of the recognized aliases match. As a result, no token metric is captured for Titan Embeddings calls.
  • bedrock-runtime-common.ts's getBedrockRuntimeOperation/buildBedrockRuntimeSpanInfo dispatch purely on AWS SDK command class name (InvokeModelCommand, etc.) and model-agnostic operation strings ("invokeModel"); there is no notion of "this invokeModel call is an embeddings call" anywhere in the plugin.

Concretely, for amazon.titan-embed-text-v2:0 (default 1024-dim, non-normalized-off) or cohere.embed-english-v3 (1024-dim, up to 96 texts per call), the full float array(s) — e.g. {"embedding": [0.0123, -0.0456, ... 1024 floats]} or {"embeddings": [[1024 floats], [1024 floats], ...], "id", "response_type", "texts"} — get logged verbatim as span output, and (for Titan) the inputTextTokenCount field is silently dropped rather than surfaced as a prompt_tokens/tokens metric.

Compare to the OpenAI plugin's embeddings handling (js/src/instrumentation/plugins/openai-plugin.ts, embeddingsCreate channel):

extractOutput: (result) => {
  const embedding = result?.data?.[0]?.embedding;
  return Array.isArray(embedding)
    ? { embedding_length: embedding.length }
    : undefined;
},

No equivalent exists for Bedrock's invokeModel embeddings path.

Braintrust docs status

not_found — https://www.braintrust.dev/docs/integrations/ai-providers/bedrock documents general Bedrock model access (chat models, tool calls, multimodal) but does not mention embeddings models or Titan/Cohere Embed specifically. A broader search of braintrust.dev/docs turned up no page describing Bedrock embeddings tracing/logging behavior. General Bedrock chat/Converse instrumentation is otherwise documented and implemented (see closed issue braintrustdata/braintrust-sdk-javascript#1741).

Upstream sources

Braintrust local repo files inspected

  • js/src/instrumentation/plugins/bedrock-runtime-plugin.ts (extractBedrockRuntimeOutput, extractBedrockRuntimeResponseMetrics, parseBedrockRuntimeMetrics)
  • js/src/instrumentation/plugins/bedrock-runtime-common.ts (getBedrockRuntimeOperation, buildBedrockRuntimeSpanInfo)
  • js/src/instrumentation/plugins/openai-plugin.ts (embeddings comparison baseline, embeddingsCreate channel)
  • js/src/instrumentation/plugins/cohere-plugin.ts, mistral-plugin.ts, groq-plugin.ts, huggingface-plugin.ts (confirmed each special-cases embeddings output minimization; grepped for SpanTypeAttribute.EMBEDDING/embedding usage across js/src/instrumentation/plugins/)
Linguagem predominante
TypeScript
Estrelas
29
Forks
22
Merge médio
1d 6h
PRs com merge (30d)
69

Preparar o ambiente

  • Inclui um Dockerfile ou arquivo Docker Compose
  • Sem modelo de pull request
  • Sem guia de contribuição

Primeiros passos

  1. Leia a issue inteira e depois o guia de contribuição do projeto.
  2. Comente na issue dizendo que vai assumir — evita que duas pessoas façam o mesmo trabalho.
  3. Faça um fork do repositório e trabalhe em uma branch.
  4. Abra um pull request que referencie o número da issue.

Mais de braintrustdata/braintrust-sdk-javascript

Todas as issues de braintrustdata/braintrust-sdk-javascript

Issues semelhantes

Mais issues de TypeScript

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.