Add LLMJudgeDetector with LLMJudgeConfig and visible failures

Aperta
  • Qualcosa non va?
#158 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
38/100
Tipo di issue
Funzionalità
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
python, pytorch
Ambito
backend, machine-learning

Direzione di ricerca

Start with lettucedetect/detectors/llm.py and the existing tests tests/test_llm_detector_pytest.py and tests/test_golden_llm_pytest.py; run the command in “Start here” to see current behavior. Read the referenced BaseDetector, judge fixtures from #129, and the contributing guidance on errors and logging before tracing prompt building, parsing, locating, and concurrency behavior. Done means the new detector passes its contract and async, error, and concurrency acceptance cases, with updated fixtures and no answer text in messages or non-content logs.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

ready-for-agent

Parent: #156 · Epic: #126 · Blocked by: #129, #143, #157
Branch from stateless_core; open the PR against it.

Problem

LLMDetector builds clients and caches from names, decides judge vs generative by substring, checks 14 keyword conflicts at runtime, and swallows failures: an unparseable reply returns no spans, an unlocated item is dropped with a debug line, and a missing fewshot_path only warns.

Current behavior

  • Constructor: lettucedetect/detectors/llm.py:63-194 (ValueError at :109, :123; missing few-shot only logs at :164).
  • _predict :395 (parse failure → logger.error at :424, no spans), _to_spans :297 (malformed item WARNING with item text at :315; unlocated item DEBUG with span text at :329), _parse_response :377.
  • Spans have no confidence unless include_reasoning is set.

Steps

  1. Add LLMJudgeConfig (frozen pydantic, extra="forbid"): temperature=0.0, lang="en", zero_shot=False, fewshot_path=None, prompt_path=None, include_reasoning=False, include_taxonomy=False (bool, list, flat dict, nested dict), min_score=0.0, verify=False, max_concurrency=30. Errors: invalid_config, invalid_language, fewshot_not_found (param config.fewshot_path). max_concurrency below 1 → invalid_config (param config.max_concurrency).
  2. Add LLMJudgeDetector(client, model, *, config=None, cache=None, model_version=None) — builds no client, reads no environment, writes no file. model_id = model.
  3. _detect for both request kinds. Reuse the prompt-building and item-locating code (first unused verbatim match, no overlaps). Score: the model's confidence with include_reasoning=True, else 1.0 plus one synthetic_score warning.
  4. Unlocated item → drop + span_not_found; item that is neither a string nor an object with text → drop + item_malformed; model-rejected item (is_hallucination false) → drop silently. Log counts and codes only; raw reply and span text only on lettucedetect.content at DEBUG.
  5. Reply that is not JSON, lacks hallucination_list, or has it as a non-list → UpstreamError (unparseable_response).
  6. adetect is inherited from BaseDetector (#141) and not overridden: the synchronous LLMClient call, the reply parsing and the locating all run inside the one worker thread. _detect keeps no per-call state on the detector, so concurrent adetect calls on one instance don't interfere.
  7. LLMJudgeConfig.min_score applies before (later) verify; the call's min_score to the final spans.
  8. Concurrency bound: the constructor creates one threading.BoundedSemaphore(config.max_concurrency) per instance. _detect takes it only around client.complete (released in a finally), never around cache access, parsing or the whole detect, so every entry point shares it and a batch pool can't deadlock on it. The verify call (#160) takes the same semaphore.
  9. Implementation note, not an acceptance criterion: write _detect as thin orchestration. Prompt building, reply parsing, item locating and verify-merging are pure functions with no client, cache or thread in their signature, shared with GenerativeDetector (#163). The only I/O is three points: the cache read, the client call (under the semaphore) and the cache write or delete. A later native async driver (epic, Out of Scope) can then reuse the pure steps. They are tested through detect and adetect only, as the epic's Testing Decisions require, not directly.

Acceptance criteria

  • Output equals the #129 judge fixtures except spans without confidence now carry 1.0 (update those fixtures as an intended change, called out in the PR).
  • All error/warning cases above have tests with a fake client; no answer text appears in any message or non-content logger.
  • await adetect(...) equals detect for a SourcesInput and a PromptInput, with and without include_reasoning, and the client call runs off the event loop's thread while the loop stays free (fake client that sleeps and records its thread).
  • Every error and warning case above also holds through adetect: UpstreamError (unparseable_response) and ConfigurationError come out of the await with the same class and code, and a caller's own client error propagates unwrapped; span_not_found/item_malformed warnings are in the result.
  • asyncio.gather over several adetect calls on one detector returns each call's own result in call order (fake client replies keyed by prompt).
  • With max_concurrency=2 and many concurrent detect and adetect calls on one detector (more than 2, started from threads and from asyncio.gather), a counting fake client never sees more than 2 calls in flight and every call gets its own result. max_concurrency=0 raises ConfigurationError (invalid_config, param config.max_concurrency).
  • LLMJudgeDetector passes the contract suite.
  • Existing tests pass; follows CONTRIBUTING "Errors and logging".

Start here

uv run --extra dev pytest tests/test_llm_detector_pytest.py tests/test_golden_llm_pytest.py -q
Lingua principale
Python
Stelle
615
Fork
60
Merge medio
1g 8h
PR unite (30g)
10

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di KRLabsOrg/LettuceDetect

Tutte le issue di KRLabsOrg/LettuceDetect

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.