Get typed spans from the encoder

クローズ
  • 何か問題がありますか？
#153 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

まだ誰も着手していません。

評価

難易度
5/5
見積もり時間
1週間以上
初心者へのやさしさ
30/100
issue の種類
機能追加
明瞭さ
明確に書かれている
活発さ
活発
技術スタック
python, pytorch
領域
backend-api-design, machine-learning

調査の方向性

Start with the encoder and detector implementations, the taxonomy head code, and the detector contract and custom taxonomy tests; the issue names no file paths, so locate these from the project structure. First read the detector contract and the existing typed golden fixtures, then trace how TransformerDetector handles taxonomy and usage. Done when typed EncoderDetector calls satisfy the acceptance criteria and the existing tests pass unchanged; the work is blocked by issue #148.

索引モデルが issue の本文から書いたものです。

説明

ready-for-agent

Parent

#126

Branch

Branch from stateless_core and open the pull request against it, not main. Close this ticket by hand once its pull request merges there: GitHub only closes issues automatically on merges into main.

What to build

An encoder detector that types its spans. The taxonomy typing head becomes a loaded object like the encoder: TaxonomyTyper takes an already loaded encoder and tokenizer, and from_pretrained is where its name becomes a model. With a typer, every span in a Detection carries category and subcategory, and the result's usage names the typer as well as the encoder, so a typed call is metered completely.

The old TransformerDetector(taxonomy_head=...) keeps working and keeps producing today's typed output.

Acceptance criteria

  • TaxonomyTyper(encoder, tokenizer, *, categories=None, subcategories=None, max_length=1024, model_id=None, model_version=None) takes loaded objects and downloads nothing; TaxonomyTyper.from_pretrained(model_id, *, device=None, categories=None, subcategories=None, model_version=None, **loader_kwargs) loads them and logs model_loaded.
  • EncoderDetector.from_pretrained(..., taxonomy_head=<id or path>, include_taxonomy=...) builds the typer through TaxonomyTyper.from_pretrained. include_taxonomy accepts True, a list of category names, a flat {name: description} dict and the nested {"categories": ..., "subcategories": ...} dict, with the same meaning as today.
  • With a typer, every DetectedSpan from detect has category and subcategory; without one both are None. For a SourcesInput the typer sees the sources joined by a newline, and for a PromptInput the prompt, as today.
  • adetect on a typed EncoderDetector returns a result equal to detect, with the same typed spans and the same two usage entries.
  • usage has two entries for an encoder with a typer: the encoder as before, and the typer with its model_id, model_version and the unit spans. When there are no spans the typer's entry reports spans as 0.
  • A typer from_pretrained for a local path takes the caller's model_version, and records the resolved commit otherwise.
  • TransformerDetector(taxonomy_head=...) still types its spans, and the output equals the typed golden fixtures.
  • The detector contract suite passes for a typed EncoderDetector.
  • The existing tests, including the custom taxonomy tests, pass unchanged.
  • The changed code follows the Errors and logging section of CONTRIBUTING.md, and no customer text is logged outside lettucedetect.content.

Blocked by

#148

User stories covered

20, 40, 50

主要言語
Python
スター
615
フォーク
60
平均マージ
2日 18時間
マージ済み PR（30日）
12

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

KRLabsOrg/LettuceDetect のほかの issue

KRLabsOrg/LettuceDetect の issue をすべて見る

似ている issue

Python の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。