Get typed spans from the encoder

Aperta
  • Qualcosa non va?
#153 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
30/100
Tipo di issue
Funzionalità
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
python, pytorch
Ambito
backend-api-design, machine-learning

Direzione di ricerca

Start with the encoder and detector implementations, the taxonomy head code, and the detector contract and custom taxonomy tests; the issue names no file paths, so locate these from the project structure. First read the detector contract and the existing typed golden fixtures, then trace how TransformerDetector handles taxonomy and usage. Done when typed EncoderDetector calls satisfy the acceptance criteria and the existing tests pass unchanged; the work is blocked by issue #148.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

ready-for-agent

Parent

#126

Branch

Branch from stateless_core and open the pull request against it, not main. Close this ticket by hand once its pull request merges there: GitHub only closes issues automatically on merges into main.

What to build

An encoder detector that types its spans. The taxonomy typing head becomes a loaded object like the encoder: TaxonomyTyper takes an already loaded encoder and tokenizer, and from_pretrained is where its name becomes a model. With a typer, every span in a Detection carries category and subcategory, and the result's usage names the typer as well as the encoder, so a typed call is metered completely.

The old TransformerDetector(taxonomy_head=...) keeps working and keeps producing today's typed output.

Acceptance criteria

  • TaxonomyTyper(encoder, tokenizer, *, categories=None, subcategories=None, max_length=1024, model_id=None, model_version=None) takes loaded objects and downloads nothing; TaxonomyTyper.from_pretrained(model_id, *, device=None, categories=None, subcategories=None, model_version=None, **loader_kwargs) loads them and logs model_loaded.
  • EncoderDetector.from_pretrained(..., taxonomy_head=<id or path>, include_taxonomy=...) builds the typer through TaxonomyTyper.from_pretrained. include_taxonomy accepts True, a list of category names, a flat {name: description} dict and the nested {"categories": ..., "subcategories": ...} dict, with the same meaning as today.
  • With a typer, every DetectedSpan from detect has category and subcategory; without one both are None. For a SourcesInput the typer sees the sources joined by a newline, and for a PromptInput the prompt, as today.
  • adetect on a typed EncoderDetector returns a result equal to detect, with the same typed spans and the same two usage entries.
  • usage has two entries for an encoder with a typer: the encoder as before, and the typer with its model_id, model_version and the unit spans. When there are no spans the typer's entry reports spans as 0.
  • A typer from_pretrained for a local path takes the caller's model_version, and records the resolved commit otherwise.
  • TransformerDetector(taxonomy_head=...) still types its spans, and the output equals the typed golden fixtures.
  • The detector contract suite passes for a typed EncoderDetector.
  • The existing tests, including the custom taxonomy tests, pass unchanged.
  • The changed code follows the Errors and logging section of CONTRIBUTING.md, and no customer text is logged outside lettucedetect.content.

Blocked by

#148

User stories covered

20, 40, 50

Lingua principale
Python
Stelle
615
Fork
60
Merge medio
1g 8h
PR unite (30g)
10

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di KRLabsOrg/LettuceDetect

Tutte le issue di KRLabsOrg/LettuceDetect

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.