Run the fine-tuned generative models through GenerativeDetector

Abierto
  • ¿Algo no está bien?
#162 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
35/100
Tipo de issue
Nueva funcionalidad
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
python
Área
ai-infra-agents

Línea de trabajo

Start by reading parent issue #126 and the blocked issues #127, #140, #144, and #156, then inspect the existing LLMJudgeDetector and LLMDetector implementations and their contract tests. Implement GenerativeDetector against the listed acceptance criteria, including cache, verification, concurrency, and the LLMDetector native path. Done means the detector contract suite and existing tests pass and the golden fixtures match.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

ready-for-agent

Parent

#126

Branch

Branch from stateless_core and open the pull request against it, not main. Close this ticket by hand once its pull request merges there: GitHub only closes issues automatically on merges into main.

What to build

A service developer serves the fine-tuned generative detectors through an OpenAI-compatible client with GenerativeDetector(client, model_id) and gets Detection results with typed spans. The detector is chosen by the class the caller builds (or by load_detector later), not by looking for substrings in the model name. Failures are visible in the same way as for the judge: an unparseable reply raises UpstreamError, and a degraded step says so in warnings.

The native path of LLMDetector delegates to GenerativeDetector from this task on.

Acceptance criteria

  • GenerativeDetector(client, model_id, *, config=None, cache=None, model_version=None) builds no client, reads no environment variable and writes no file (apart from a cache the caller passed).
  • GenerativeConfig has the fields and defaults in the epic, and has no zero_shot, fewshot_path or prompt_path: passing one raises ConfigurationError (invalid_config) with its param. An unknown language raises it with invalid_language.
  • detect and adetect accept a SourcesInput and a PromptInput, use the frozen generative prompt, and return typed spans: category and subcategory from the reply, and reasoning from the reply's explanation with include_reasoning=True. Spans are located in the answer as for the judge, and text equals the answer between the offsets. The output matches the golden fixtures once spans carry score 1.0.
  • The same per-instance semaphore as in LLMJudgeDetector wraps each client call (the verify call included). max_concurrency below 1 raises ConfigurationError (invalid_config, param config.max_concurrency), and a counting fake client never sees more than max_concurrency calls in flight.
  • Every span has score 1.0 and the Detection has exactly one synthetic_score warning.
  • The custom taxonomy forms (True, a list of names, a flat dict, the nested dict) work as today.
  • An item that can't be located is dropped with span_not_found, and a malformed item with item_malformed.
  • A reply that is not JSON, lacks hallucinated_spans or has it as something other than a list raises UpstreamError (unparseable_response).
  • With verify=True the verify pass behaves as in LLMJudgeDetector, including verification_skipped.
  • With a cache, the keys come from make_cache_key (kinds "native" and "verify") with model_version; a hit makes no client call; a reply is written only after it parses, and a cached entry that doesn't parse is deleted and raises UpstreamError (unparseable_response).
  • usage has one entry with model_id, model_version and the units llm_calls, cache_hits, source_chars and answer_chars.
  • GenerativeDetector passes the detector contract suite with a fake client.
  • The native path of LLMDetector delegates to GenerativeDetector and its predict* output equals the golden fixtures, apart from spans that had no confidence now carrying 1.0.
  • The existing tests pass unchanged.
  • The changed code follows the Errors and logging section of CONTRIBUTING.md, and no customer text is logged outside lettucedetect.content.

Blocked by

#127, #140, #144, #156

User stories covered

3, 4, 5, 7, 8, 17, 19, 21, 23, 24, 26, 29, 34, 40, 50, 57

Lenguaje dominante
Python
Estrellas
615
Forks
60
Merge medio
2 d 18 h
PR fusionados (30 d)
12

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de KRLabsOrg/LettuceDetect

Todos los issues de KRLabsOrg/LettuceDetect

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.