Run the fine-tuned generative models through GenerativeDetector

オープン
  • 何か問題がありますか？
#162 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

まだ誰も着手していません。

評価

難易度
5/5
見積もり時間
1週間以上
初心者へのやさしさ
35/100
issue の種類
機能追加
明瞭さ
おおむね明確
活発さ
活発
技術スタック
python
領域
ai-infra-agents

調査の方向性

Start by reading parent issue #126 and the blocked issues #127, #140, #144, and #156, then inspect the existing LLMJudgeDetector and LLMDetector implementations and their contract tests. Implement GenerativeDetector against the listed acceptance criteria, including cache, verification, concurrency, and the LLMDetector native path. Done means the detector contract suite and existing tests pass and the golden fixtures match.

索引モデルが issue の本文から書いたものです。

説明

ready-for-agent

Parent

#126

Branch

Branch from stateless_core and open the pull request against it, not main. Close this ticket by hand once its pull request merges there: GitHub only closes issues automatically on merges into main.

What to build

A service developer serves the fine-tuned generative detectors through an OpenAI-compatible client with GenerativeDetector(client, model_id) and gets Detection results with typed spans. The detector is chosen by the class the caller builds (or by load_detector later), not by looking for substrings in the model name. Failures are visible in the same way as for the judge: an unparseable reply raises UpstreamError, and a degraded step says so in warnings.

The native path of LLMDetector delegates to GenerativeDetector from this task on.

Acceptance criteria

  • GenerativeDetector(client, model_id, *, config=None, cache=None, model_version=None) builds no client, reads no environment variable and writes no file (apart from a cache the caller passed).
  • GenerativeConfig has the fields and defaults in the epic, and has no zero_shot, fewshot_path or prompt_path: passing one raises ConfigurationError (invalid_config) with its param. An unknown language raises it with invalid_language.
  • detect and adetect accept a SourcesInput and a PromptInput, use the frozen generative prompt, and return typed spans: category and subcategory from the reply, and reasoning from the reply's explanation with include_reasoning=True. Spans are located in the answer as for the judge, and text equals the answer between the offsets. The output matches the golden fixtures once spans carry score 1.0.
  • The same per-instance semaphore as in LLMJudgeDetector wraps each client call (the verify call included). max_concurrency below 1 raises ConfigurationError (invalid_config, param config.max_concurrency), and a counting fake client never sees more than max_concurrency calls in flight.
  • Every span has score 1.0 and the Detection has exactly one synthetic_score warning.
  • The custom taxonomy forms (True, a list of names, a flat dict, the nested dict) work as today.
  • An item that can't be located is dropped with span_not_found, and a malformed item with item_malformed.
  • A reply that is not JSON, lacks hallucinated_spans or has it as something other than a list raises UpstreamError (unparseable_response).
  • With verify=True the verify pass behaves as in LLMJudgeDetector, including verification_skipped.
  • With a cache, the keys come from make_cache_key (kinds "native" and "verify") with model_version; a hit makes no client call; a reply is written only after it parses, and a cached entry that doesn't parse is deleted and raises UpstreamError (unparseable_response).
  • usage has one entry with model_id, model_version and the units llm_calls, cache_hits, source_chars and answer_chars.
  • GenerativeDetector passes the detector contract suite with a fake client.
  • The native path of LLMDetector delegates to GenerativeDetector and its predict* output equals the golden fixtures, apart from spans that had no confidence now carrying 1.0.
  • The existing tests pass unchanged.
  • The changed code follows the Errors and logging section of CONTRIBUTING.md, and no customer text is logged outside lettucedetect.content.

Blocked by

#127, #140, #144, #156

User stories covered

3, 4, 5, 7, 8, 17, 19, 21, 23, 24, 26, 29, 34, 40, 50, 57

主要言語
Python
スター
615
フォーク
60
平均マージ
2日 18時間
マージ済み PR（30日）
12

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

KRLabsOrg/LettuceDetect のほかの issue

KRLabsOrg/LettuceDetect の issue をすべて見る

似ている issue

Python の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。