[bot] LiteLLM: Batches API passthrough (`create_batch`/`retrieve_batch`/`list_batches`/`cancel_batch`) not instrumented

Ouverte
  • Quelque chose ne va pas ?
#1,063 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub

Les mainteneurs répondent en général sous 1 jour

Personne n'a encore pris cette issue.

Évaluation

Difficulté
4/5
Temps estimé
3-5 jours
Accessibilité débutants
38/100
Type d'issue
Fonctionnalité
Clarté
Plutôt claire
Activité
Active
Stack technique
python
Domaine
observability

Piste de recherche

Start with py/src/braintrust/integrations/litellm/patchers.py, where each existing LiteLLM entry point (for example the embedding patcher) is registered in _ALL_LITELLM_PATCHERS, then read the matching wrapper in tracing.py. The batch functions need a span shape for asynchronous jobs that span several calls, so decide that design with maintainers first. Done means create, retrieve, list and cancel (sync and async) each produce a span, with tests built on the existing cassette setup.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Description

python

Summary

LiteLLM ships a documented Batches API passthrough surface — litellm.create_batch / litellm.acreate_batch, litellm.retrieve_batch / litellm.aretrieve_batch, litellm.list_batches / litellm.alist_batches, and litellm.cancel_batch / litellm.acancel_batch — which proxies real async batch-inference jobs to the underlying provider's Batch API (OpenAI, Azure OpenAI, Vertex AI, Bedrock, Mistral, vLLM, xAI). This is a distinct execution surface from litellm.batch_completion (client-side fan-out over concurrent completion() calls, tracked separately and closed not_planned in #755 because it already produces spans via the patched completion function).

The Braintrust LiteLLM integration (py/src/braintrust/integrations/litellm/) instruments 22 patchers across 11 top-level entry points (completion/acompletion, text_completion/atext_completion, responses/aresponses, decisions/adecisions (+ systemone/asystemone aliases), image_generation/aimage_generation, embedding/aembedding, moderation/amoderation, speech/aspeech, transcription/atranscription, rerank/arerank), but has no patcher for any of the 8 Batches API functions. Calls made through create_batch/retrieve_batch/list_batches/cancel_batch (or their async variants) produce zero Braintrust spans — no batch submission metadata, no status, no linkage to the resulting generations.

What is missing

LiteLLM function Description Instrumented?
litellm.completion / .acompletion Single-call chat completion Yes
litellm.create_batch / .acreate_batch Submit an async batch inference job to the underlying provider's Batch API No
litellm.retrieve_batch / .aretrieve_batch Poll batch job status/result metadata No
litellm.list_batches / .alist_batches List batch jobs No
litellm.cancel_batch / .acancel_batch Cancel a batch job No

Braintrust docs status

not_found. The Braintrust LiteLLM integration page (https://www.braintrust.dev/docs/integrations/sdk-integrations/litellm) explicitly enumerates the entry points it patches — completion, acompletion, text_completion, atext_completion, responses, aresponses, image_generation, aimage_generation, embedding, aembedding, moderation, amoderation, speech, aspeech, transcription, atranscription, rerank, arerank — with no mention of create_batch/retrieve_batch/list_batches/cancel_batch anywhere on the page.

Upstream sources

  • LiteLLM Batches API docs: https://docs.litellm.ai/docs/batches — documents create_batch/acreate_batch, retrieve_batch/aretrieve_batch, list_batches/alist_batches, cancel_batch/acancel_batch as current, actively maintained functions proxying to OpenAI, Azure, Vertex AI, Bedrock, Mistral, vLLM, and xAI batch endpoints.
  • Confirmed actively maintained via recent upstream PRs, e.g. fix(vertex_ai): honor timeout on batch create, retrieve, list and cancel (BerriAI/litellm#47322) and fix(batches): register ownership for every batch create path (BerriAI/litellm#39810).

Local repo files inspected

  • py/src/braintrust/integrations/litellm/patchers.py — defines 22 patchers (LiteLLMCompletionPatcher through LiteLLMArerankPatcher); no batch patcher exists.
  • py/src/braintrust/integrations/litellm/tracing.py — wrapper implementations for all currently-patched entry points; no batch wrapper functions.
  • py/src/braintrust/integrations/litellm/integration.py — LiteLLMIntegration.patchers is sourced entirely from _ALL_LITELLM_PATCHERS in patchers.py.
  • py/src/braintrust/wrappers/litellm.py — grep for "batch" (case-insensitive): zero matches.
  • Grep for "batch" (case-insensitive) across py/src/braintrust/integrations/litellm/: only one incidental match inside a cassette YAML (unrelated content string); no code or test references to create_batch/retrieve_batch/list_batches/cancel_batch.

Relationship to existing issues

  • #755 (closed, not_planned) covers litellm.batch_completion/batch_completion_models specifically — a client-side concurrent-fan-out helper that already produces spans via the patched completion() function. This issue is about the separate, real async Batches API passthrough (create_batch et al.), which has no instrumentation path today since it doesn't call completion() internally.
Langage dominant
Python
Étoiles
22
Forks
30
Merge moyen
9 h
PR mergées (30 j)
128

Préparer son environnement

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Autres issues de braintrustdata/braintrust-sdk-python

Toutes les issues de braintrustdata/braintrust-sdk-python

Issues similaires

Plus d'issues Python

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.