Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

[bot] Add Replicate Python SDK native integration for run, stream, and predictions execution instrumentation (345,748 weekly downloads)

Abierto
  • ¿Algo no está bien?
#516 1 comentario 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
45/100
Tipo de issue
Nueva funcionalidad
Claridad
Bastante claro
Estado de actividad
Tranquilo
Stack tecnológico
python
Área
ai, devtools

Línea de trabajo

Comienza con las integraciones existentes en py/src/braintrust/integrations/, los wrappers en py/src/braintrust/wrappers/ y los puntos de registro en py/src/braintrust/auto.py, integrations/init.py, versioning.py, py/noxfile.py y pyproject.toml. Revisa cómo las integraciones de proveedores existentes gestionan las API de run, stream, async y prediction antes de definir la integración de Replicate. La implementación debe cubrir las superficies de ejecución indicadas, las pruebas, el registro y los campos y métricas de span propuestos.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

bot-automation new-integration

Summary

The Replicate Python SDK (replicate) is the official Python client for Replicate, a platform for running AI models (LLMs, image generation, video, audio, and more) in the cloud. Its replicate.run() and replicate.predictions.create() APIs are the primary execution surfaces. The latest release is v1.0.7 (May 27, 2026). This repository has zero native instrumentation for any replicate SDK execution surface — no integration directory, no wrapper, no patcher, no auto_instrument() support.

Braintrust documents a gateway/proxy-based integration with Replicate where users route Replicate calls through the Braintrust gateway using an OpenAI-compatible client. This approach does not cover users of the native replicate Python SDK, which has its own distinct API (replicate.run("owner/model", input={...})). The replicate SDK is not an openai.OpenAI subclass and cannot be wrapped with wrap_openai().

The same pattern exists for other providers in this repo: Mistral, OpenRouter, xAI, Groq, and Together AI all have OpenAI-compatible endpoints, yet received dedicated native integrations because users following official provider documentation and pip install <provider> get zero Braintrust tracing from the OpenAI wrapper alone.

What needs to be instrumented

The replicate package (v1.0.7) exposes these execution surfaces, none of which are instrumented:

Model execution (highest priority)
API Description Return type
replicate.run(model, input, ...) Synchronously run a model and return the complete output Any (model-specific: str, list, FileOutput, ...)
replicate.stream(model, input, ...) Stream model output as server-sent events — primarily for LLM token streaming Iterator[ServerSentEvent]

replicate.run() accepts any Replicate model identifier (e.g., "meta/meta-llama-3-8b-instruct", "stability-ai/sdxl") and an input dict with model-specific parameters. The model identifier encodes the provider and model name — useful for span metadata.

LLM streaming: For language models, replicate.run() returns an iterator yielding text tokens. replicate.stream() provides SSE-based streaming for the same use case.

Async execution
API Description Return type
await replicate.async_run(model, input, ...) Async equivalent of replicate.run() Any
Background predictions
API Description Return type
replicate.predictions.create(model, input, ...) Create a prediction without waiting for it to complete Prediction
await replicate.predictions.async_create(model, input, ...) Async prediction creation Prediction
prediction.wait() Wait for a prediction to complete Prediction

Prediction objects expose .status, .output, .logs, .metrics (includes predict_time), .urls (cancel, get, stream webhooks).

Implementation notes

Generic execution model: Unlike OpenAI or Anthropic where model names map to a known schema, Replicate model inputs and outputs are model-specific. The integration should log the model identifier (owner/model or owner/model:version), the input dict, and the output value. Token count extraction is only possible for models that return usage metadata in their response or logs.

Model identifier: Replicate model identifiers like "meta/meta-llama-3-8b-instruct" or "stability-ai/stable-diffusion-3" encode the model type. The span should capture model (full identifier) and provider: "replicate" in metadata.

Output types: LLM models return iterators of text tokens; image models return FileOutput objects; other models may return strings, numbers, lists, or dicts. The integration should handle all cases, collecting token text for LLM outputs and logging output metadata (type, size if file) for others.

Streaming LLM output: replicate.run() for LLM models returns a generator. The span should be created before iteration starts and finalized (with full output text) when iteration completes. replicate.stream() follows the same pattern with SSE events.

predict_time metric: Prediction.metrics["predict_time"] provides server-side inference time in seconds — a useful span metric.

Client pattern: The module-level functions (replicate.run()) delegate to a default Client instance. Patching Client.run, Client.stream, and Client.async_run covers both usage patterns.

Authentication: Uses REPLICATE_API_TOKEN env var or explicit api_token kwarg. VCR cassettes need Authorization: Token <key> header sanitization.

Proposed span shape

Span field Content
input model identifier, input dict
output collected text output (LLM), null or file metadata (image/video)
metadata provider: "replicate", model (full identifier), version if specified
metrics predict_time from Prediction.metrics, time_to_first_token (streaming LLMs)

No coverage in any instrumentation layer

  • No integration directory (py/src/braintrust/integrations/replicate/)
  • No wrapper function (e.g. wrap_replicate())
  • No patcher in any existing integration
  • No nox test session (test_replicate)
  • No version entry in py/src/braintrust/integrations/versioning.py
  • No mention in py/src/braintrust/integrations/__init__.py

A grep for replicate across py/src/braintrust/ returns zero matches.

Braintrust docs status

unclear — The Braintrust Replicate integration page documents a gateway/proxy approach using an OpenAI client pointed at the Braintrust gateway. This covers users who access Replicate through the OpenAI-compatible gateway endpoint but does not cover native replicate SDK users who call replicate.run() directly. There is no auto_instrument() reference and no wrap_replicate() function documented.

Upstream references

Local repo files inspected

  • py/src/braintrust/integrations/ — no replicate/ directory exists on main
  • py/src/braintrust/wrappers/ — no Replicate wrapper
  • py/noxfile.py — no test_replicate session
  • py/src/braintrust/integrations/__init__.py — Replicate not listed in integration registry
  • py/src/braintrust/integrations/versioning.py — no Replicate version matrix
  • py/pyproject.toml [tool.braintrust.matrix] — no Replicate entry
  • py/src/braintrust/auto.py — Replicate not listed in auto_instrument() parameters
  • Full repo grep for replicate across py/src/braintrust/ — zero matches
Lenguaje dominante
Python
Estrellas
20
Forks
18
Merge medio
21 h 8 min
PR fusionados (30 d)
81

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de braintrustdata/braintrust-sdk-python

Todos los issues de braintrustdata/braintrust-sdk-python

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.