Load an EncoderDetector explicitly and call detect

Cerrado
  • ¿Algo no está bien?
#148 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
25/100
Tipo de issue
Nueva funcionalidad
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
python, pytorch
Área
backend, machine-learning

Línea de trabajo

Start by reading the parent issue #126 and the blocked issues #127 and #140, then inspect the existing TransformerDetector and its tests for the named predict methods and golden fixtures. Implement the EncoderDetector API, config validation, loading, scoring, truncation warnings, usage records, and thread-safe async behavior while preserving the compatibility requirements. Done means the acceptance criteria pass, including the detector contract suite and existing tests.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

ready-for-agent

Parent

#126

Branch

Branch from stateless_core and open the pull request against it, not main. Close this ticket by hand once its pull request merges there: GitHub only closes issues automatically on merges into main.

What to build

A service developer loads the fine-tuned encoder once at startup with EncoderDetector.from_pretrained, or builds EncoderDetector from a model and tokenizer it already holds, and then calls detect and adetect with a SourcesInput or a PromptInput. The result is a Detection with scored spans, optional token scores, a usage record naming the exact weights, and a warning when the input was cut to fit max_length.

TransformerDetector becomes a deprecated subclass of EncoderDetector, so there is one implementation, and today's predict* output, attributes and private method names stay as they are. Typed spans (the TaxonomyTyper) and batch methods come in later tasks; until then TransformerDetector with taxonomy_head still loads its typer as today and predict_prompt_batch is unchanged.

Acceptance criteria

  • EncoderDetector(model, tokenizer, typer=None, *, config=None, model_id=None, model_version=None) downloads nothing, reads no environment variable and writes no file: constructing it with a stub model under offline settings in an empty working directory succeeds and leaves the directory empty.
  • EncoderConfig has max_length=4096, batch_size=16 and lang="en". An unknown option raises ConfigurationError (invalid_config), an unknown language raises it with invalid_language, and a batch_size below 1 raises it with invalid_batch_size, each with its param.
  • EncoderDetector.from_pretrained(model_id, *, device=None, taxonomy_head=None, include_taxonomy=True, model_version=None, config=None, **loader_kwargs) loads the tokenizer and the model and logs model_loaded. model_version is the resolved Hugging Face commit; for a local path it is the caller's model_version, or None when none is given. (taxonomy_head is accepted but typed spans are task #153.)
  • detect and adetect accept a SourcesInput and a PromptInput and return a Detection whose spans equal today's predict(output_format="spans") and predict_prompt(output_format="spans") output for the same input (golden fixtures), with score equal to today's confidence and text equal to the answer between the offsets. Sources that don't fit are scored in passage groups and aggregated with max(), as today.
  • include_tokens=True returns tokens as TokenScore items (token text, offsets into the answer, score, flagged) with the same tokens and probabilities as today's "tokens" output; the tokens are not filtered by min_score.
  • When the tokenized prompt and answer exceed max_length, or a passage group still exceeds it, the Detection has the warning input_truncated, and a WARNING record with its code and the token counts is logged (never the text).
  • usage has one entry with model_id, model_version and the units sources (omitted for a PromptInput), source_chars, answer_chars and input_tokens.
  • EncoderDetector passes the detector contract suite from task #140.
  • EncoderDetector holds one private threading.RLock, created on first use under a module-level guard lock (not in __init__), around its whole scoring path, with no config field. asyncio.gather over many adetect calls with different inputs on one detector, using the real offline tokenizer from conftest.py, returns each call's own result, equal to a sequential detect, and a stub model shows no two scoring calls overlapping.
  • TransformerDetector(model_path=..., max_length, device, lang, taxonomy_head, include_taxonomy, batch_size, **tok_kwargs) is a subclass of EncoderDetector, warns LettuceDetectDeprecationWarning naming EncoderDetector.from_pretrained, keeps its attributes (model, tokenizer, device, max_length, lang, batch_size, typer) and its private method names, and its predict and predict_prompt output equals the golden fixtures.
  • TransformerDetector also exposes detect and adetect (inherited), returning a result equal to an EncoderDetector built from the same model, with no warning beyond the one at construction.
  • A TransformerDetector created with __new__ and only model, tokenizer, device, max_length, lang and typer set by hand still answers predict with spans.
  • A TransformerDetector created with __new__ answers concurrent calls too: racing first calls create exactly one lock.
  • The existing tests pass unchanged.
  • The changed code follows the Errors and logging section of CONTRIBUTING.md, and no customer text is logged outside lettucedetect.content.

Blocked by

#127, #140

User stories covered

3, 4, 5, 7, 12, 15, 16, 18, 25, 34, 40, 47, 56, 57

Lenguaje dominante
Python
Estrellas
615
Forks
60
Merge medio
2 d 18 h
PR fusionados (30 d)
12

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de KRLabsOrg/LettuceDetect

Todos los issues de KRLabsOrg/LettuceDetect

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.