Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

Clarify STOP authority on the native-token fast path without changing rendering

Aperta
  • Qualcosa non va?
#961 3 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
54/100
Tipo di issue
Documentazione
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
python
Ambito
ai, backend-api-design

Direzione di ricerca

Read _stop_suffix and _tokenize_history in src/art/trajectories/_tokenize.py, then inspect the no-tokenizer test at tests/unit/trajectories/test_tokenize.py and the process path in _parallel.py. Document that native-only output with finish_reason="stop" does not prove a missing STOP token, while preserving rendering and the no-import contract; verify the cited reproduction and existing tests.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

On main d734e25fca0c343ed06a5fa491a15736ff954cf3, complete native Chat Completions token IDs can be tokenized without a tokenizer, while STOP remains unidentified for a materialized response with finish_reason="stop" and no integer stop_reason. Supplying the correct EOS authority changes only the terminal STOP bit in the example below.

This is an authority/API gap, not evidence that marking with a known tokenizer is wrong. Native IDs and finish_reason="stop" alone do not identify which token, if any, belongs to the terminating sequence. In this mode an absent STOP bit must not be interpreted as proof that the token is not a stop.

Minimal public reproduction (no model or tokenizer download):

from datetime import UTC, datetime
from openai.types.chat import ChatCompletion
import art.trajectories as tr

class StopAuthority:
    eos_token_id = 9
    def __call__(self, text, **kwargs):
        assert text == "END"
        return [8, 9]

now = datetime(2026, 1, 1, tzinfo=UTC)
response = ChatCompletion.model_validate({
    "id": "public-stop-example", "object": "chat.completion", "created": 0,
    "model": "public/example", "choices": [{
        "index": 0, "finish_reason": "stop", "stop_reason": None,
        "message": {"role": "assistant", "content": "answer"},
        "prompt_token_ids": [1], "token_ids": [2, 9],
        "logprobs": {"content": [
            {"token": f"token_id:{t}", "logprob": -0.5,
             "bytes": [], "top_logprobs": []} for t in [2, 9]
        ]},
    }],
})
trajectory = tr.Trajectory(exchanges=tr.TrajectoryExchanges(chat_completions=[
    tr.ChatCompletionsExchange(
        request={"model": "public/example", "messages": [
            {"role": "user", "content": "question"}
        ]}, response=response, start_time=now, end_time=now,
    )
]))
unknown = trajectory.tokenize()
known = trajectory.tokenize(tokenizer=StopAuthority())
assert unknown.tokens == known.tokens == [1, 2, 9]
assert [int(f) for f in unknown.flags] == [1, 23, 23]
assert [int(f) for f in known.flags] == [1, 23, 31]

The minimal authority object intentionally has no renderer. The checked fixture also verifies identical logprobs and SAMPLED first-occurrence masks. This does not establish equivalence for consumers that select STOP or for rendering with a full tokenizer.

Relevant current contracts:

  • _stop_suffix returns zero without a tokenizer unless an exact non-bool integer stop reason proves the final token. _sampled_stop_suffix also retains the separate no-materialized-output case.
  • _tokenize_history passes tokenizer=None into the exact native builder before resolving any renderer. This preserves the intentional no-import contract tested by test_exact_tokens_form_one_append_only_history_without_tokenizer, including W&B artifact model names.
  • Passing a full tokenizer is not a general render-neutral workaround: _history_needs_synthetic_stop and other renderer-selection predicates inspect it.
  • Parallel tokenization's process path explicitly supplies tokenizer=None; explicit tokenizer objects currently select the thread path. A future separate authority must survive that boundary and remain bound to the correct model, rather than silently being omitted or shared across unrelated models.

A small safe next step is documenting the incomplete-STOP meaning of the native-only path. If callers need complete STOP flags without changing render selection, consider an explicit, per-model STOP authority distinct from the renderer tokenizer, carried through public/parallel APIs into suffix/marker checks only. Do not eagerly load metadata, guess EOS IDs, mark every final sampled token, or change synthetic-stop rendering to resolve this gap.

Validation: the real ART/Pydantic reproduction and eight existing tests passed on the pinned source (EOS/tool stop, string stop, explicit native integer stop, metadata-only output, native-only no-load, and length-stop rendering). Local execution bypassed only top-level ART package startup and disabled heavy imports/network; no model, provider, GPU, or frozen-run source was used or changed. No production patch accompanies this issue.

Lingua principale
Python
Stelle
10.8k
Fork
1k
Merge medio
16h 14m
PR unite (30g)
148

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di OpenPipe/ART

Tutte le issue di OpenPipe/ART

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.