[bot] LiteLLM: Batches API passthrough (`create_batch`/`retrieve_batch`/`list_batches`/`cancel_batch`) not instrumented

Abierto
  • ¿Algo no está bien?
#1,063 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
38/100
Tipo de issue
Nueva funcionalidad
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
python
Área
observability

Línea de trabajo

Start with py/src/braintrust/integrations/litellm/patchers.py, where each existing LiteLLM entry point (for example the embedding patcher) is registered in _ALL_LITELLM_PATCHERS, then read the matching wrapper in tracing.py. The batch functions need a span shape for asynchronous jobs that span several calls, so decide that design with maintainers first. Done means create, retrieve, list and cancel (sync and async) each produce a span, with tests built on the existing cassette setup.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

python

Summary

LiteLLM ships a documented Batches API passthrough surface — litellm.create_batch / litellm.acreate_batch, litellm.retrieve_batch / litellm.aretrieve_batch, litellm.list_batches / litellm.alist_batches, and litellm.cancel_batch / litellm.acancel_batch — which proxies real async batch-inference jobs to the underlying provider's Batch API (OpenAI, Azure OpenAI, Vertex AI, Bedrock, Mistral, vLLM, xAI). This is a distinct execution surface from litellm.batch_completion (client-side fan-out over concurrent completion() calls, tracked separately and closed not_planned in #755 because it already produces spans via the patched completion function).

The Braintrust LiteLLM integration (py/src/braintrust/integrations/litellm/) instruments 22 patchers across 11 top-level entry points (completion/acompletion, text_completion/atext_completion, responses/aresponses, decisions/adecisions (+ systemone/asystemone aliases), image_generation/aimage_generation, embedding/aembedding, moderation/amoderation, speech/aspeech, transcription/atranscription, rerank/arerank), but has no patcher for any of the 8 Batches API functions. Calls made through create_batch/retrieve_batch/list_batches/cancel_batch (or their async variants) produce zero Braintrust spans — no batch submission metadata, no status, no linkage to the resulting generations.

What is missing

LiteLLM function Description Instrumented?
litellm.completion / .acompletion Single-call chat completion Yes
litellm.create_batch / .acreate_batch Submit an async batch inference job to the underlying provider's Batch API No
litellm.retrieve_batch / .aretrieve_batch Poll batch job status/result metadata No
litellm.list_batches / .alist_batches List batch jobs No
litellm.cancel_batch / .acancel_batch Cancel a batch job No

Braintrust docs status

not_found. The Braintrust LiteLLM integration page (https://www.braintrust.dev/docs/integrations/sdk-integrations/litellm) explicitly enumerates the entry points it patches — completion, acompletion, text_completion, atext_completion, responses, aresponses, image_generation, aimage_generation, embedding, aembedding, moderation, amoderation, speech, aspeech, transcription, atranscription, rerank, arerank — with no mention of create_batch/retrieve_batch/list_batches/cancel_batch anywhere on the page.

Upstream sources

  • LiteLLM Batches API docs: https://docs.litellm.ai/docs/batches — documents create_batch/acreate_batch, retrieve_batch/aretrieve_batch, list_batches/alist_batches, cancel_batch/acancel_batch as current, actively maintained functions proxying to OpenAI, Azure, Vertex AI, Bedrock, Mistral, vLLM, and xAI batch endpoints.
  • Confirmed actively maintained via recent upstream PRs, e.g. fix(vertex_ai): honor timeout on batch create, retrieve, list and cancel (BerriAI/litellm#47322) and fix(batches): register ownership for every batch create path (BerriAI/litellm#39810).

Local repo files inspected

  • py/src/braintrust/integrations/litellm/patchers.py — defines 22 patchers (LiteLLMCompletionPatcher through LiteLLMArerankPatcher); no batch patcher exists.
  • py/src/braintrust/integrations/litellm/tracing.py — wrapper implementations for all currently-patched entry points; no batch wrapper functions.
  • py/src/braintrust/integrations/litellm/integration.py — LiteLLMIntegration.patchers is sourced entirely from _ALL_LITELLM_PATCHERS in patchers.py.
  • py/src/braintrust/wrappers/litellm.py — grep for "batch" (case-insensitive): zero matches.
  • Grep for "batch" (case-insensitive) across py/src/braintrust/integrations/litellm/: only one incidental match inside a cassette YAML (unrelated content string); no code or test references to create_batch/retrieve_batch/list_batches/cancel_batch.

Relationship to existing issues

  • #755 (closed, not_planned) covers litellm.batch_completion/batch_completion_models specifically — a client-side concurrent-fan-out helper that already produces spans via the patched completion() function. This issue is about the separate, real async Batches API passthrough (create_batch et al.), which has no instrumentation path today since it doesn't call completion() internally.
Lenguaje dominante
Python
Estrellas
22
Forks
30
Merge medio
9 h
PR fusionados (30 d)
128

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de braintrustdata/braintrust-sdk-python

Todos los issues de braintrustdata/braintrust-sdk-python

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.