[Bug]: tokenize emits a non-existent new line token for empty input

Open

#1,133 opened on Sep 8, 2026 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty: 3/5
Estimated time: 1-2 days
Newbie friendliness: 74/100

Issue type: Bug
Clarity: Clearly specified
Activity status: Active
Tech stack: python
Domain: backend

Research direction

Start by running the repro.py example with GraalPy and CPython, then trace the GraalPy implementation of _tokenize.TokenizerIter used by tokenize.generate_tokens. The fix is done when empty input produces only the CPython-compatible ENDMARKER with matching positions, while the existing comment and newline cases remain correct.

Written by the indexing model from the issue text.

Description

bug

Describe the bug

When using token.generate_tokens() on an empty string, a NL token is returned when no such token exists for CPython. The return token also spans a non-existent length, which caused stubgen-pyx to crash. Only effects "", # ... and "\n" were fine. This caused stubgen-pyx to crash on an otherwise empty file.

Likely occurs in GraalPy's re-implementation of _tokenize.TokenizerIter

Operating system

Linux

CPU architecture

x86_64

GraalPy version

GraalPy 3.13.14 (GraalVM CE Native 25.3.4.1)

JDK version

openjdk 25.0.4 2026-07-21 LTS (Temurin-25.0.4+7)

Context configuration

N/A

Steps to reproduce

# repro.py
import io, tokenize
for tok in tokenize.generate_tokens(io.StringIO("").readline):
    print(tok)

$ graalpy repro.py
TokenInfo(type=63 (NL), string='', start=(1, 0), end=(1, 1), line='')
TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')

$ python3.13 repro.py          # and python3.14, identical
TokenInfo(type=0 (ENDMARKER), string='', start=(1, 0), end=(1, 0), line='')

Native version

import io, _tokenize
for t in _tokenize.TokenizerIter(io.StringIO("").readline, extra_tokens=True):
    print(t)
#GraalPy 3.13.14:  (63, '', (1, 0), (1, 1), '')      ← phantom NL
#                  (0,  '', (2, 0), (2, 0), '')
#CPython 3.13:     (0,  '', (1, 0), (1, 0), '')

Expected behavior

Should match the same output as CPython 3.13, and not emit non-existent spans.

Stack trace

N/A

Additional context

N/A

Dominant language: Python
Stars: 1.6k
Forks: 155
Avg merge: 8h 20m
Merged PRs (30d): 43

Contributor guide

Open the contributing guide

First steps

Read the whole issue, then the project's contributing guide.
Comment on the issue to say you are picking it up — it saves two people doing the same work.
Fork the repository and make your change on a branch.
Open a pull request that references the issue number.

Similar issues

slack-gif-creator: validate_gif() multiplies the last frame's duration by the frame count, so reported duration/fps are wrong (1.0s GIF reported as 1.8s) Open

Difficulty 2/5 1-3 hours Newbie friendliness 75/100

anthropics/skills#1811 · 1 comment · Opened Sep 23, 2026
App fails to start with gradio 6 (allowed by `gradio>=5.49.1`) Open

Difficulty 2/5 1-3 hours Newbie friendliness 75/100

speaches-ai/speaches#678 · Opened Sep 23, 2026
ModuleNotFoundError: No module named 'mcp.server.fastmcp'. Open

bug

Difficulty 2/5 1-3 hours Newbie friendliness 75/100

datalayer/mcp-compose#42 · Opened Sep 23, 2026
import spacy fails with ModuleNotFoundError: No module named 'click' Open

Difficulty 2/5 1-3 hours Newbie friendliness 75/100

conda-forge/spacy-feedstock#177 · Opened Sep 23, 2026
humaneval: extract_function_body corrupts a fenced body-only reply, so the same correct code scores INCORRECT fenced and CORRECT unfenced Open

Difficulty 2/5 1-3 hours Newbie friendliness 70/100

UKGovernmentBEIS/inspect_evals#2523 · Opened Sep 23, 2026

More Python issues