Run the fine-tuned generative models through GenerativeDetector

Aperta
  • Qualcosa non va?
#162 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
35/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
python
Ambito
ai-infra-agents

Direzione di ricerca

Start by reading parent issue #126 and the blocked issues #127, #140, #144, and #156, then inspect the existing LLMJudgeDetector and LLMDetector implementations and their contract tests. Implement GenerativeDetector against the listed acceptance criteria, including cache, verification, concurrency, and the LLMDetector native path. Done means the detector contract suite and existing tests pass and the golden fixtures match.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

ready-for-agent

Parent

#126

Branch

Branch from stateless_core and open the pull request against it, not main. Close this ticket by hand once its pull request merges there: GitHub only closes issues automatically on merges into main.

What to build

A service developer serves the fine-tuned generative detectors through an OpenAI-compatible client with GenerativeDetector(client, model_id) and gets Detection results with typed spans. The detector is chosen by the class the caller builds (or by load_detector later), not by looking for substrings in the model name. Failures are visible in the same way as for the judge: an unparseable reply raises UpstreamError, and a degraded step says so in warnings.

The native path of LLMDetector delegates to GenerativeDetector from this task on.

Acceptance criteria

  • GenerativeDetector(client, model_id, *, config=None, cache=None, model_version=None) builds no client, reads no environment variable and writes no file (apart from a cache the caller passed).
  • GenerativeConfig has the fields and defaults in the epic, and has no zero_shot, fewshot_path or prompt_path: passing one raises ConfigurationError (invalid_config) with its param. An unknown language raises it with invalid_language.
  • detect and adetect accept a SourcesInput and a PromptInput, use the frozen generative prompt, and return typed spans: category and subcategory from the reply, and reasoning from the reply's explanation with include_reasoning=True. Spans are located in the answer as for the judge, and text equals the answer between the offsets. The output matches the golden fixtures once spans carry score 1.0.
  • The same per-instance semaphore as in LLMJudgeDetector wraps each client call (the verify call included). max_concurrency below 1 raises ConfigurationError (invalid_config, param config.max_concurrency), and a counting fake client never sees more than max_concurrency calls in flight.
  • Every span has score 1.0 and the Detection has exactly one synthetic_score warning.
  • The custom taxonomy forms (True, a list of names, a flat dict, the nested dict) work as today.
  • An item that can't be located is dropped with span_not_found, and a malformed item with item_malformed.
  • A reply that is not JSON, lacks hallucinated_spans or has it as something other than a list raises UpstreamError (unparseable_response).
  • With verify=True the verify pass behaves as in LLMJudgeDetector, including verification_skipped.
  • With a cache, the keys come from make_cache_key (kinds "native" and "verify") with model_version; a hit makes no client call; a reply is written only after it parses, and a cached entry that doesn't parse is deleted and raises UpstreamError (unparseable_response).
  • usage has one entry with model_id, model_version and the units llm_calls, cache_hits, source_chars and answer_chars.
  • GenerativeDetector passes the detector contract suite with a fake client.
  • The native path of LLMDetector delegates to GenerativeDetector and its predict* output equals the golden fixtures, apart from spans that had no confidence now carrying 1.0.
  • The existing tests pass unchanged.
  • The changed code follows the Errors and logging section of CONTRIBUTING.md, and no customer text is logged outside lettucedetect.content.

Blocked by

#127, #140, #144, #156

User stories covered

3, 4, 5, 7, 8, 17, 19, 21, 23, 24, 26, 29, 34, 40, 50, 57

Lingua principale
Python
Stelle
615
Fork
60
Merge medio
1g 8h
PR unite (30g)
10

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di KRLabsOrg/LettuceDetect

Tutte le issue di KRLabsOrg/LettuceDetect

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.