Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Clarify STOP authority on the native-token fast path without changing rendering

Open
  • Is something wrong?
#961 3 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 3 days

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
54/100
Issue type
Documentation
Clarity
Mostly clear
Activity status
Active
Tech stack
python
Domain
ai, backend-api-design

Research direction

Read _stop_suffix and _tokenize_history in src/art/trajectories/_tokenize.py, then inspect the no-tokenizer test at tests/unit/trajectories/test_tokenize.py and the process path in _parallel.py. Document that native-only output with finish_reason="stop" does not prove a missing STOP token, while preserving rendering and the no-import contract; verify the cited reproduction and existing tests.

Written by the indexing model from the issue text.

Description

On main d734e25fca0c343ed06a5fa491a15736ff954cf3, complete native Chat Completions token IDs can be tokenized without a tokenizer, while STOP remains unidentified for a materialized response with finish_reason="stop" and no integer stop_reason. Supplying the correct EOS authority changes only the terminal STOP bit in the example below.

This is an authority/API gap, not evidence that marking with a known tokenizer is wrong. Native IDs and finish_reason="stop" alone do not identify which token, if any, belongs to the terminating sequence. In this mode an absent STOP bit must not be interpreted as proof that the token is not a stop.

Minimal public reproduction (no model or tokenizer download):

from datetime import UTC, datetime
from openai.types.chat import ChatCompletion
import art.trajectories as tr

class StopAuthority:
    eos_token_id = 9
    def __call__(self, text, **kwargs):
        assert text == "END"
        return [8, 9]

now = datetime(2026, 1, 1, tzinfo=UTC)
response = ChatCompletion.model_validate({
    "id": "public-stop-example", "object": "chat.completion", "created": 0,
    "model": "public/example", "choices": [{
        "index": 0, "finish_reason": "stop", "stop_reason": None,
        "message": {"role": "assistant", "content": "answer"},
        "prompt_token_ids": [1], "token_ids": [2, 9],
        "logprobs": {"content": [
            {"token": f"token_id:{t}", "logprob": -0.5,
             "bytes": [], "top_logprobs": []} for t in [2, 9]
        ]},
    }],
})
trajectory = tr.Trajectory(exchanges=tr.TrajectoryExchanges(chat_completions=[
    tr.ChatCompletionsExchange(
        request={"model": "public/example", "messages": [
            {"role": "user", "content": "question"}
        ]}, response=response, start_time=now, end_time=now,
    )
]))
unknown = trajectory.tokenize()
known = trajectory.tokenize(tokenizer=StopAuthority())
assert unknown.tokens == known.tokens == [1, 2, 9]
assert [int(f) for f in unknown.flags] == [1, 23, 23]
assert [int(f) for f in known.flags] == [1, 23, 31]

The minimal authority object intentionally has no renderer. The checked fixture also verifies identical logprobs and SAMPLED first-occurrence masks. This does not establish equivalence for consumers that select STOP or for rendering with a full tokenizer.

Relevant current contracts:

  • _stop_suffix returns zero without a tokenizer unless an exact non-bool integer stop reason proves the final token. _sampled_stop_suffix also retains the separate no-materialized-output case.
  • _tokenize_history passes tokenizer=None into the exact native builder before resolving any renderer. This preserves the intentional no-import contract tested by test_exact_tokens_form_one_append_only_history_without_tokenizer, including W&B artifact model names.
  • Passing a full tokenizer is not a general render-neutral workaround: _history_needs_synthetic_stop and other renderer-selection predicates inspect it.
  • Parallel tokenization's process path explicitly supplies tokenizer=None; explicit tokenizer objects currently select the thread path. A future separate authority must survive that boundary and remain bound to the correct model, rather than silently being omitted or shared across unrelated models.

A small safe next step is documenting the incomplete-STOP meaning of the native-only path. If callers need complete STOP flags without changing render selection, consider an explicit, per-model STOP authority distinct from the renderer tokenizer, carried through public/parallel APIs into suffix/marker checks only. Do not eagerly load metadata, guess EOS IDs, mark every final sampled token, or change synthetic-stop rendering to resolve this gap.

Validation: the real ART/Pydantic reproduction and eight existing tests passed on the pinned source (EOS/tool stop, string stop, explicit native integer stop, metadata-only output, native-only no-load, and length-stop rendering). Local execution bypassed only top-level ART package startup and disabled heavy imports/network; no model, provider, GPU, or frozen-run source was used or changed. No production patch accompanies this issue.

Dominant language
Python
Stars
10.8k
Forks
989
Avg merge
11h 38m
Merged PRs (30d)
104

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from OpenPipe/ART

All issues in OpenPipe/ART

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.