[bot] AWS Bedrock InvokeModel for embeddings not instrumented
Nessuno ha ancora preso questa issue.
Valutazione
- Difficoltà
- 4/5
- Tempo stimato
- 3-5 giorni
- Idoneità per principianti
- 68/100
- Tipo di issue
- Funzionalità
- Chiarezza
- Abbastanza chiara
- Stato di attività
- Tranquilla
- Stack tecnologico
- aws, java
- Ambito
- cloud, observability
Direzione di ricerca
Inizia da BraintrustBedrockInterceptor.java e BraintrustAWSBedrock.java per tracciare come vengono selezionate le operazioni Converse, quindi leggi InstrumentationSemConv.java per esaminare il tagging esistente delle richieste e delle risposte Bedrock. Esamina BraintrustAWSBedrockTest.java e aggiungi la copertura per InvokeModelRequest usando payload strutturati come quelli di Titan e Cohere. Il lavoro è completato quando le chiamate di embedding creano span con metadati del modello/provider, l’input acquisito, il tipo di embedding e le metriche dei token specificate.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Descrizione
Summary
The AWS Bedrock instrumentation in BraintrustBedrockInterceptor explicitly skips all operations other than Converse and ConverseStream. This means embedding model calls — which use the InvokeModel API, not Converse — produce no spans, no input capture, and no token metrics.
All embedding models on AWS Bedrock require InvokeModel:
- Amazon Titan Text Embeddings v2 (
amazon.titan-embed-text-v2:0) - Amazon Titan Text Embeddings v1 (
amazon.titan-embed-text-v1) - Amazon Titan Multimodal Embeddings (
amazon.titan-embed-image-v1) - Cohere Embed English v3 (
cohere.embed-english-v3) - Cohere Embed Multilingual v3 (
cohere.embed-multilingual-v3)
None of these models support the Converse API — InvokeModel is the only invocation path for Bedrock embeddings.
What is missing
BraintrustBedrockInterceptor.beforeExecution() (lines 60–71) explicitly limits instrumentation to two operations:
private static final Set<String> INSTRUMENTED_OPERATIONS = Set.of("Converse", "ConverseStream");
// Only instrument Converse and ConverseStream — other Bedrock operations
// (InvokeModel, ApplyGuardrail, etc.) are not LLM calls we know how to tag.
if (!INSTRUMENTED_OPERATIONS.contains(operationName)) {
return;
}
When a user calls client.invokeModel(...) with an embedding model (e.g. amazon.titan-embed-text-v2:0), the interceptor returns immediately without creating a span. No span is created, no input text is captured, and no embedding output or token metrics are recorded.
The BraintrustAWSBedrock.wrap() Javadoc on the sync builder explicitly states it traces "every converse call" and the async builder traces "every converseStream call" — embedding calls via InvokeModel are not in scope at all today.
A minimal InvokeModel embedding span should capture:
braintrust.metadata: model ID (from the request URI path/model/{modelId}/invoke), providerbraintrust.input_json: theinputText(or equivalent input field) from the request bodybraintrust.metrics:prompt_tokensfrom the responseinputTextTokenCount(Titan) ormeta.inputTokenCount(Cohere)braintrust.span_attributes:{"type": "embedding"}
Braintrust docs status
- AWS Bedrock listed as a supported cloud provider at https://www.braintrust.dev/docs/integrations/ai-providers: supported (for generative use)
- No mention of Bedrock embeddings or
InvokeModelinstrumentation in any Braintrust docs page: not_found - For comparison, OpenAI and Google GenAI embedding instrumentation exists in this repo (though with detail gaps tracked in #66 and #65)
Upstream sources
- AWS Bedrock InvokeModel API reference: https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_InvokeModel.html — "You use model inference to generate text, images, and embeddings."
- Amazon Titan Text Embeddings v2 example: https://docs.aws.amazon.com/bedrock/latest/userguide/bedrock-runtime_example_bedrock-runtime_InvokeModel_TitanTextEmbeddings_section.html — uses
client.invokeModel()with model IDamazon.titan-embed-text-v2:0; response includesembeddingarray andinputTextTokenCount - Cohere Embed on Bedrock: https://docs.aws.amazon.com/bedrock/latest/userguide/model-parameters-embed.html — uses
InvokeModelwithtextsarray; response includesembeddingsandmeta.inputTokenCount - AWS SDK for Java v2 BedrockRuntimeClient:
invokeModel(InvokeModelRequest)— operation nameInvokeModel, same SDK client as Converse
Local files inspected
braintrust-sdk/instrumentation/aws_bedrock_2_30_0/src/main/java/dev/braintrust/instrumentation/awsbedrock/v2_30_0/BraintrustBedrockInterceptor.java— line 60:Set.of("Converse", "ConverseStream"); lines 63–71:beforeExecutionreturns without span for any other operation name includingInvokeModelbraintrust-sdk/instrumentation/aws_bedrock_2_30_0/src/main/java/dev/braintrust/instrumentation/awsbedrock/v2_30_0/BraintrustAWSBedrock.java— Javadoc on bothwrap()overloads confirms only Converse/ConverseStream are traced; noinvokeModelwrapping existsbraintrust-sdk/instrumentation/aws_bedrock_2_30_0/src/test/java/dev/braintrust/instrumentation/awsbedrock/v2_30_0/BraintrustAWSBedrockTest.java— all tests useConverseRequestandConverseStreamRequest; noInvokeModelRequesttest existsbraintrust-sdk/src/main/java/dev/braintrust/instrumentation/InstrumentationSemConv.java—tagBedrockRequest()andtagBedrockResponse()only handle Converse-shaped JSON (messages, output.message, usage.inputTokens); InvokeModel request/response bodies have entirely different schemas
- Lingua principale
- Java
- Stelle
- 21
- Fork
- 5
- Merge medio
- 2g 7h
- PR unite (30g)
- 8
Guida per i contributori
Apri la guida per i contributori
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Altre issue di braintrustdata/braintrust-sdk-java
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 82/100
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 82/100
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 82/100
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 78/100
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 68/100
Tutte le issue di braintrustdata/braintrust-sdk-java
Issue simili
-
executions.Query — startDate and timeRange filters are sent with inverted comparison operators Apertaarea/plugin
Difficoltà 2/5 1-3 ore Idoneità per principianti 75/100
kestra-io/plugin-kestra#190 ·
-
litertlm-android AAR ships no consumer ProGuard rules → "mid == null" SIGABRT in minified apps Aperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 70/100
google-ai-edge/LiteRT-LM#3739 ·
-
bug
Difficoltà 2/5 1-3 ore Idoneità per principianti 75/100
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 75/100
integra-team-red/meet-map#249 ·
-
[Studio][Bug] Cancelled create-user dialog keeps the password and admin switch for the next attempt Aperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 75/100
apache/rocketmq-dashboard#5064 ·