Add token scores, input_truncated and usage to EncoderDetector

Aperta
  • Qualcosa non va?
#150 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
55/100
Tipo di issue
Funzionalità
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
python, pytorch
Ambito
backend-api-design, machine-learning

Direzione di ricerca

Start with transformer.py at _decode_sample, _predict_single, the batch and predict_prompt truncation paths, and the passage-group path; run the focused pytest command in the issue. Compare token results with the #128 golden output and check usage and truncation behavior, including a log-capture test that verifies counts without input text. The issue is blocked by #149 and asks for a branch from stateless_core.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

ready-for-agent

Parent: #148 · Epic: #126 · Blocked by: #149
Branch from stateless_core; open the PR against it.

Problem

The encoder can show token-level evidence only through the output_format="tokens" dict path, truncation is only logged (and not at all when a passage group still exceeds max_length), and nothing reports which weights ran or how much work a call did.

Current behavior

  • Tokens: transformer.py:163 _decode_sample, :241 _predict_single with output_format="tokens" (pred, prob keys).
  • Truncation warnings: transformer.py:319-322 (batch path, text-free but only logged) and :511-521 (predict_prompt, only logged). The passage-group path (:105-125) logs nothing when a group still exceeds max_length.
  • No usage.

Steps

  1. include_tokens=True (set supports_tokens = True) returns Detection.tokens as TokenScore (token text, offsets into the answer, score = model probability, flagged = old pred). Same tokens and probabilities as the old "tokens" output; not filtered by min_score; aggregated over passage groups with max().
  2. When the tokenized prompt + answer exceed max_length, or a group still exceeds it, add the warning input_truncated to the Detection and log a WARNING with the code and token counts (never text).
  3. Add usage: one entry with model_id, model_version, units sources (omitted for PromptInput), source_chars, answer_chars, input_tokens (summed over passage groups).

Acceptance criteria

  • Tokens equal the #128 golden "tokens" output field for field (pred→flagged, prob→score).
  • A too-long input yields input_truncated; a log-capture test shows the code and counts and no input text.
  • usage has the listed units; sources absent for PromptInput.
  • Contract suite still passes; existing tests pass; follows CONTRIBUTING "Errors and logging".

Start here

uv run --extra dev pytest tests/test_inference_pytest.py -q -k "truncat or token"
Lingua principale
Python
Stelle
615
Fork
60
Merge medio
1g 8h
PR unite (30g)
10

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di KRLabsOrg/LettuceDetect

Tutte le issue di KRLabsOrg/LettuceDetect

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.