Run many requests with detect_batch and adetect_batch

Aperta
  • Qualcosa non va?
#175 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
30/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
python, pytorch
Ambito
backend, machine-learning

Direzione di ricerca

Start with the batch result type and failure reporting decision referenced in #126, then review the detector contract and the blocked work in #148, #156, #162, #166, and #169. Implement and check the batch contract across each detector, including concurrency, encoder batching, and the predict_prompt_batch adapters. Done means the listed acceptance criteria and existing tests pass.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

ready-for-agent

Parent

#126

Branch

Branch from stateless_core and open the pull request against it, not main. Close this ticket by hand once its pull request merges there: GitHub only closes issues automatically on merges into main.

What to build

A service developer sends a list of DetectRequest items to any detector and gets one result per request in input order, with the encoder scoring them in padded batches and the LLM-backed detectors running them concurrently under a limit the caller sets. One failed request doesn't throw away the results of the others, reported in the way the batch result decision defines.

The old predict_prompt_batch on every detector becomes an adapter over detect_batch. It keeps today's output, and it keeps raising on the first failure, as today.

Acceptance criteria

  • detect_batch(requests, *, min_score=0.0, include_tokens=False, include_detail=False) and adetect_batch exist on every detector, take any mix of SourcesInput and PromptInput, and return one result per request in input order. An empty sequence returns an empty result.
  • The result type and the way a failed request is reported are the ones the batch result decision defines. Whatever it defines, the results of requests that succeeded are returned alongside the failures.
  • EncoderDetector scores requests in padded groups of config.batch_size in one forward pass per group, and each result equals the result of a single detect call for the same request (including usage for that request).
  • Gathered adetect calls and adetect_batch calls on one EncoderDetector, single and padded-batch mixed, return the same results as the same requests run one by one (the encoder serialises scoring with its own lock, see task #148).
  • LLMJudgeDetector and GenerativeDetector run requests concurrently with at most config.max_concurrency client calls in flight per detector instance (max_concurrency defaults to 30), counted across detect, adetect, detect_batch and adetect_batch together, and each result equals the single detect result.
  • RAGFactCheckerDetector runs its requests through the same concurrent path, bounded by its max_concurrency keyword on concurrent detect calls.
  • include_tokens=True and include_detail=True raise InvalidInputError for the whole batch on detectors that don't support them, before any request runs.
  • adetect_batch is a coroutine function that runs detect_batch in a worker thread and returns an equal result (results, order, per-request failures). It is inherited by every detector, including those that override detect_batch.
  • adetect_batch leaves the event loop free while it waits. Its whole-batch errors (unsupported flags) come from the awaited call before any request runs, an empty sequence returns an empty result, and max_concurrency still bounds in-flight calls, also across gathered adetect_batch calls and a batch running next to single detect or adetect calls.
  • The detector contract suite gains a batch section that runs for every detector: order, empty input, equal to single calls, adetect_batch equal to detect_batch, and the failure reporting once decided.
  • predict_prompt_batch on TransformerDetector, LLMDetector and RAGFactCheckerDetector calls detect_batch and returns today's output (golden fixtures); a failed request raises the first error, as today.
  • The existing tests pass unchanged.
  • The changed code follows the Errors and logging section of CONTRIBUTING.md, and no customer text is logged outside lettucedetect.content.

Blocked by

#148, #156, #162, #166, #169, and the decision elsewhere on the batch result type and failure reporting (see Further Notes in #126).

User stories covered

35, 36, 37, 38, 40, 57

Lingua principale
Python
Stelle
615
Fork
60
Merge medio
1g 8h
PR unite (30g)
10

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di KRLabsOrg/LettuceDetect

Tutte le issue di KRLabsOrg/LettuceDetect

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.