Add RAGFactCheckerDetector.detect with injected fact checker and merge fix

Aperta
  • Qualcosa non va?
#170 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
52/100
Tipo di issue
Funzionalità
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
python
Ambito
backend-api-design, testing

Direzione di ricerca

Start with tests/test_golden_rag_fact_checker_pytest.py and lettucedetect/detectors/rag_fact_checker.py; first run the provided pytest command and inspect the #130 merge fixture. Implement the injected fact checker path, detection outputs, span text correction, and shared concurrency bound described in the steps. Done means the acceptance criteria pass, including async and batch behavior, without reading the environment on the injected path.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

ready-for-agent

Parent: #169 · Epic: #126 · Blocked by: #130, #143
Branch from stateless_core; open the PR against it.

Problem

RAGFactCheckerDetector can't be tested or used without building RAGFactChecker from an API key, ignores misspelt keyword arguments, has no detect, and can return a span whose text is not the text at its offsets.

Current behavior

  • lettucedetect/detectors/rag_fact_checker.py:17-40 constructor builds RAGFactChecker and swallows **kwargs.
  • :155 _convert_to_spans: confidence 0.9 (:192) and 0.7 for partial matches (:211). :220 _merge_overlapping_spans sets end = max(...) (:237) but keeps the first span's text.

Steps

  1. Constructor becomes RAGFactCheckerDetector(openai_api_key=None, model="gpt-4o", base_url=None, temperature=0.0, *, fact_checker=None, max_concurrency=30, **kwargs). With fact_checker, nothing is built and the environment isn't read; without it, build RAGFactChecker as today (document this exception: RAGFactChecker is outside this package). model_id = model.
  2. Implement _detect for SourcesInput and PromptInput. score is today's confidence (0.9 full, 0.7 partial); add one synthetic_score warning when there is any span.
  3. After merging, set the span's text to answer[start:end]. Update the merge fixture from #130 as an intended change, called out in the PR.
  4. include_tokens=True raises InvalidInputError (tokens_not_supported).
  5. usage: one entry, model_version=None, units sources, source_chars, answer_chars.
  6. max_concurrency is a named keyword (so **kwargs doesn't swallow it); below 1 raises ConfigurationError (invalid_config, param max_concurrency). The constructor creates one threading.BoundedSemaphore(max_concurrency), and _detect holds it for the whole fact-check call, because the LLM calls happen inside RAGFactChecker and can't be counted from outside. detect, adetect and the batch methods therefore share the bound on concurrent detect calls.

Acceptance criteria

  • detect spans equal #130 "spans" output with score = old confidence; merged-span text equals the answer between offsets.
  • await adetect(...) equals detect (inherited, in a worker thread); an injected fact checker is called from that thread only.
  • With max_concurrency=2 and many concurrent detect and adetect calls on one detector, a counting fake fact checker is never called by more than 2 at once; max_concurrency=0 raises ConfigurationError (invalid_config, param max_concurrency); the default is 30.
  • Passes the contract suite with a fake fact checker; the injected path touches no environment.
  • Existing tests pass; follows CONTRIBUTING "Errors and logging".

Start here

uv run --extra dev pytest tests/test_golden_rag_fact_checker_pytest.py -q
Lingua principale
Python
Stelle
615
Fork
60
Merge medio
1g 8h
PR unite (30g)
10

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di KRLabsOrg/LettuceDetect

Tutte le issue di KRLabsOrg/LettuceDetect

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.