Add detect_batch and adetect_batch with concurrent execution

Aperta
  • Qualcosa non va?
#176 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
35/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
python
Ambito
api, backend, testing

Direzione di ricerca

Start with BaseDetector in base.py:47, then read the batch implementations in lettucedetect/detectors/llm.py:614 and lettucedetect/detectors/rag_fact_checker.py:102, along with the referenced issues and batch decision. Run uv run --extra dev pytest tests/test_detector_contract_pytest.py -q to inspect the existing contract tests. Done means batch calls preserve order and single-call results, report per-request failures, obey concurrency limits, and pass the stated contract and acceptance checks.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

ready-for-agent

Parent: #175 · Epic: #126 · Blocked by: #161, #165, #171, and the decision elsewhere on the batch result type and failure reporting
Branch from stateless_core; open the PR against it.

Problem

LLMDetector.predict_prompt_batch runs a hard-coded pool of 30 threads and raises on the first failure, dropping the results of requests that succeeded. No other detector has a real batch method.

Current behavior

  • lettucedetect/detectors/llm.py:614 ThreadPoolExecutor(max_workers=30).
  • lettucedetect/detectors/rag_fact_checker.py:102 predict_prompt_batch is a plain loop.
  • base.py:47 predict_prompt_batch is abstract.

Steps

  1. Add detect_batch(requests, *, min_score=0.0, include_tokens=False, include_detail=False) and adetect_batch to BaseDetector. One result per request in input order; an empty sequence returns an empty result. Result type and per-request failure reporting follow the batch decision, and successes are returned alongside failures.
  2. The default implementation runs detect concurrently in a pool of max_concurrency threads (config.max_concurrency, default 30, for LLMJudgeDetector and GenerativeDetector; the constructor keyword max_concurrency, default 30, for RAGFactCheckerDetector, from #170). The pool only limits how many requests of one batch are submitted at once. The limit across calls is the per-instance semaphore each detector already has (#158, #163, #170), which every detect goes through.
  3. async def adetect_batch(requests, *, min_score=0.0, include_tokens=False, include_detail=False) runs detect_batch in a worker thread (asyncio.to_thread) and returns the same result type. Detectors that override detect_batch (for example #177) get adetect_batch without changes; none overrides it. Whole-batch errors (step 4) are raised from the awaited call; per-request failures are reported as the batch decision says, identically to detect_batch. The pool of step 2 runs inside that one worker thread. Because the bound comes from the per-instance semaphore and not from the pool, two gathered adetect_batch calls, or a batch running next to single detect and adetect calls, together stay within max_concurrency client calls in flight.
  4. include_tokens / include_detail unsupported → InvalidInputError for the whole batch before any request runs. Messages refer to requests[i] by position, never text.
  5. Add the batch section to the contract suite: order, empty input, equal to single calls, failure reporting.

Acceptance criteria

  • At most max_concurrency client calls in flight on one detector instance (test with a counting fake client), for detect_batch and for adetect_batch, counted across entry points: two adetect_batch calls gathered together, and a batch running next to single detect and adetect calls. For RAGFactCheckerDetector the counted unit is a concurrent detect call.
  • A batch with many more requests than max_concurrency (for example 100 requests with max_concurrency=2) completes with every result correct and no deadlock.
  • adetect_batch equals detect_batch for the same requests (results, order, per-request failures), and the event loop keeps running while it awaits (loop ticker test).
  • Cancelling adetect_batch mid-batch leaves the detector usable: no deadlock, the worker pool shuts down once its in-flight calls finish, no exception reaches the loop, and the next detect_batch returns the right results.
  • adetect_batch([]) returns an empty result; unsupported flags raise the same InvalidInputError from the awaited call before any request runs.
  • Each result equals the single detect result; order preserved.
  • One failing request doesn't discard the others, reported per the decision.
  • Contract suite batch section passes for every detector class that exists at this point.
  • Existing tests pass; follows CONTRIBUTING "Errors and logging".

Start here

uv run --extra dev pytest tests/test_detector_contract_pytest.py -q
Lingua principale
Python
Stelle
615
Fork
60
Merge medio
1g 8h
PR unite (30g)
10

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di KRLabsOrg/LettuceDetect

Tutte le issue di KRLabsOrg/LettuceDetect

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.