Add GenerativeDetector with GenerativeConfig and typed spans

Aperta
  • Qualcosa non va?
#163 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
48/100
Tipo di issue
Funzionalità
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
python
Ambito
ai, backend-api-design

Direzione di ricerca

Start with lettucedetect/detectors/llm.py and run uv run --extra dev pytest tests/test_llm_detector_pytest.py -q -k native. Follow the shared parsing, locating, cache, and concurrency code described in the issue and inspect the #129 native fixtures. Done means the new detector matches those fixtures with scored spans, supports both request kinds, and passes the listed configuration, error, and concurrency criteria.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

ready-for-agent

Parent: #162 · Epic: #126 · Blocked by: #158, #129
Branch from stateless_core; open the PR against it.

Problem

The fine-tuned generative models are served by the same class as the judge, chosen by a substring in the model name, with spans that never carry a score and an empty or garbage reply read as "no hallucination found".

Current behavior

  • lettucedetect/detectors/llm.py:26 _looks_native(model) guesses from the name; :121-125 raises ValueError if native is combined with zero_shot, fewshot_path or prompt_path.
  • llm.py:437 _predict_native uses the frozen generative prompt, then gen.spans_to_offsets(answer, gen.parse_spans(cached or "")) (:479): a reply without JSON or without hallucinated_spans yields no spans.

Steps

  1. Add GenerativeConfig (frozen pydantic, extra="forbid"): temperature=0.0, lang="en" (used by the verify prompt), include_reasoning=False, include_taxonomy=False (same four forms), verify=False, max_concurrency=30. zero_shot, fewshot_path, prompt_path are not fields → ConfigurationError (invalid_config) with param; unknown language → invalid_language. max_concurrency below 1 → invalid_config (param config.max_concurrency).
  2. Add GenerativeDetector(client, model_id, *, config=None, cache=None, model_version=None) — builds no client, reads no environment, writes no file.
  3. _detect for both request kinds with the frozen prompt. Spans are DetectedSpan with category/subcategory from the reply and reasoning from explanation when include_reasoning=True. Located in the answer as the judge does (share the helper).
  4. Every span has score 1.0 and the Detection has exactly one synthetic_score warning.
  5. Unlocated item → span_not_found; malformed item → item_malformed. Reply that is not JSON, lacks hallucinated_spans, or has a non-list → UpstreamError (unparseable_response).
  6. Concurrency bound: the same per-instance threading.BoundedSemaphore(config.max_concurrency) as #158 step 8, taken only around each client call (share the code). The verify call (#164) takes it too.
  7. Same implementation note as #158 step 9: keep _detect thin, with the pure parsing, locating and verify-merging steps shared with the judge and I/O only at the cache read, the client call and the cache write or delete.

Acceptance criteria

  • Output equals the #129 native fixtures except spans carry score 1.0 (intended change, in the PR description).
  • await adetect(...) equals detect for both request kinds, with adetect inherited and the same errors coming out of the await.
  • With max_concurrency=2 and many concurrent detect and adetect calls on one detector, a counting fake client never sees more than 2 calls in flight; max_concurrency=0 raises ConfigurationError (invalid_config, param config.max_concurrency).
  • All custom taxonomy forms work; passing a judge-only option raises as above.
  • No answer text in messages or non-content loggers.
  • Contract suite passes with a fake client; existing tests pass; follows CONTRIBUTING "Errors and logging".

Start here

uv run --extra dev pytest tests/test_llm_detector_pytest.py -q -k native
Lingua principale
Python
Stelle
615
Fork
60
Merge medio
2g 18h
PR unite (30g)
12

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di KRLabsOrg/LettuceDetect

Tutte le issue di KRLabsOrg/LettuceDetect

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.