Run the LLM judge through LLMJudgeDetector

Aperta
  • Qualcosa non va?
#156 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
30/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
python
Ambito
ai, backend-api-design

Direzione di ricerca

Start by reading the parent issue #126, the blocked issues #127, #140 and #144, and the Errors and logging section of CONTRIBUTING.md. Then inspect the existing LLMDetector judge path, client implementations, cache helpers, and detector contract and golden tests to map the required behavior. Done means LLMJudgeDetector passes the stated contract, the judge path delegates to it, and the existing tests pass.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

ready-for-agent

Parent

#126

Branch

Branch from stateless_core and open the pull request against it, not main. Close this ticket by hand once its pull request merges there: GitHub only closes issues automatically on merges into main.

What to build

A service developer passes an LLM client and a model id to LLMJudgeDetector and gets Detection results from the zero-shot or few-shot judge, with reasoning, taxonomy and the verify pass as config options. A failed call is visible: an unparseable reply raises UpstreamError instead of reading as "no hallucination found", the clients raise typed errors, and every degraded step (an unlocated item, a skipped verification) says so in warnings. The detector caches only when the caller passes a ResponseCache.

The judge path of LLMDetector delegates to LLMJudgeDetector from this task on, so there is one implementation; LLMDetector's own warnings and argument handling come in task #166.

Acceptance criteria

  • LLMJudgeDetector(client, model, *, config=None, cache=None, model_version=None) builds no client, reads no environment variable and writes no file (apart from a cache the caller passed); its model_id is model.
  • LLMJudgeConfig has the fields and defaults in the epic. An unknown option raises ConfigurationError (invalid_config), an unknown language raises it with invalid_language, and a fewshot_path that doesn't exist raises it with fewshot_not_found, each with its param.
  • detect and adetect accept a SourcesInput and a PromptInput and locate each item in the answer as today (first unused verbatim match, no overlaps). text equals the answer between the offsets, and the output matches the golden fixtures once spans without confidence carry score 1.0.
  • adetect is inherited, not overridden: it returns a result equal to detect for both request kinds, runs the client call, the cache access and the verify pass in one worker thread off the event loop, raises the same errors (UpstreamError, ConfigurationError) from the awaited call, and gathered adetect calls on one detector return their own results in call order.
  • One threading.BoundedSemaphore(config.max_concurrency) per detector instance wraps each client call (the verify call included) and nothing else. With max_concurrency=2 and many concurrent detect and adetect calls, a counting fake client never sees more than 2 calls in flight and every call gets its own result. max_concurrency below 1 raises ConfigurationError (invalid_config, param config.max_concurrency).
  • score is the model's confidence with include_reasoning=True, and 1.0 otherwise, in which case the Detection has exactly one synthetic_score warning. category, subcategory and reasoning are set as today, every include_taxonomy form works, and LLMJudgeConfig.min_score is applied before the verify pass while the call's min_score is applied to the final spans.
  • An item that can't be located is dropped with the warning span_not_found, and an item that is neither a string nor an object with text is dropped with item_malformed. An item the model rejected itself is dropped without a warning. No answer text appears in a message or on a module logger.
  • A reply that is not JSON, lacks hallucination_list or has it as something other than a list raises UpstreamError (unparseable_response).
  • With verify=True rejected spans are dropped; a verification reply that can't be parsed keeps the first-pass spans, adds verification_skipped and logs fallback_used with that code at WARNING.
  • OpenAIClient raises ConfigurationError (authentication_failed, quota_exceeded) for authentication and quota errors and UpstreamError (upstream_unavailable) for timeouts, connection errors, rate limits and server errors, chained with raise ... from. BedrockClient raises UpstreamError (empty_response) when the reply has no tool call. A caller's own LLMClient errors propagate unwrapped.
  • The detector uses the cache's sync methods from the worker thread, so adetect has the same hit, write-after-parse and delete-on-unparseable behaviour as detect, and concurrent adetect calls lose no cache key.
  • With a cache, the keys come from make_cache_key (kinds "judge" and "verify") with model_version; a hit makes no client call; a reply is written only after it parses; a cached entry that doesn't parse is deleted and the call raises UpstreamError (unparseable_response).
  • usage has one entry with model_id, model_version and the units llm_calls (calls actually made, the verify call included), cache_hits, source_chars and answer_chars.
  • LLMJudgeDetector passes the detector contract suite with a fake client.
  • The judge path of LLMDetector delegates to LLMJudgeDetector and its predict* output equals the golden fixtures, apart from spans that had no confidence now carrying 1.0 (a recorded, intended change).
  • The existing tests pass unchanged.
  • The changed code follows the Errors and logging section of CONTRIBUTING.md, and no customer text is logged outside lettucedetect.content (the raw reply included).

Blocked by

#127, #140, #144

User stories covered

3, 4, 5, 7, 8, 17, 19, 21, 22, 23, 24, 26, 29, 34, 40, 47, 50, 57

Lingua principale
Python
Stelle
615
Fork
60
Merge medio
1g 8h
PR unite (30g)
10

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di KRLabsOrg/LettuceDetect

Tutte le issue di KRLabsOrg/LettuceDetect

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.