Get typed spans from the encoder

Abierto
  • ¿Algo no está bien?
#153 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
30/100
Tipo de issue
Nueva funcionalidad
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
python, pytorch
Área
backend-api-design, machine-learning

Línea de trabajo

Start with the encoder and detector implementations, the taxonomy head code, and the detector contract and custom taxonomy tests; the issue names no file paths, so locate these from the project structure. First read the detector contract and the existing typed golden fixtures, then trace how TransformerDetector handles taxonomy and usage. Done when typed EncoderDetector calls satisfy the acceptance criteria and the existing tests pass unchanged; the work is blocked by issue #148.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

ready-for-agent

Parent

#126

Branch

Branch from stateless_core and open the pull request against it, not main. Close this ticket by hand once its pull request merges there: GitHub only closes issues automatically on merges into main.

What to build

An encoder detector that types its spans. The taxonomy typing head becomes a loaded object like the encoder: TaxonomyTyper takes an already loaded encoder and tokenizer, and from_pretrained is where its name becomes a model. With a typer, every span in a Detection carries category and subcategory, and the result's usage names the typer as well as the encoder, so a typed call is metered completely.

The old TransformerDetector(taxonomy_head=...) keeps working and keeps producing today's typed output.

Acceptance criteria

  • TaxonomyTyper(encoder, tokenizer, *, categories=None, subcategories=None, max_length=1024, model_id=None, model_version=None) takes loaded objects and downloads nothing; TaxonomyTyper.from_pretrained(model_id, *, device=None, categories=None, subcategories=None, model_version=None, **loader_kwargs) loads them and logs model_loaded.
  • EncoderDetector.from_pretrained(..., taxonomy_head=<id or path>, include_taxonomy=...) builds the typer through TaxonomyTyper.from_pretrained. include_taxonomy accepts True, a list of category names, a flat {name: description} dict and the nested {"categories": ..., "subcategories": ...} dict, with the same meaning as today.
  • With a typer, every DetectedSpan from detect has category and subcategory; without one both are None. For a SourcesInput the typer sees the sources joined by a newline, and for a PromptInput the prompt, as today.
  • adetect on a typed EncoderDetector returns a result equal to detect, with the same typed spans and the same two usage entries.
  • usage has two entries for an encoder with a typer: the encoder as before, and the typer with its model_id, model_version and the unit spans. When there are no spans the typer's entry reports spans as 0.
  • A typer from_pretrained for a local path takes the caller's model_version, and records the resolved commit otherwise.
  • TransformerDetector(taxonomy_head=...) still types its spans, and the output equals the typed golden fixtures.
  • The detector contract suite passes for a typed EncoderDetector.
  • The existing tests, including the custom taxonomy tests, pass unchanged.
  • The changed code follows the Errors and logging section of CONTRIBUTING.md, and no customer text is logged outside lettucedetect.content.

Blocked by

#148

User stories covered

20, 40, 50

Lenguaje dominante
Python
Estrellas
615
Forks
60
Merge medio
1 d 8 h
PR fusionados (30 d)
10

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de KRLabsOrg/LettuceDetect

Todos los issues de KRLabsOrg/LettuceDetect

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.