BUG: bpb underestimated when tokenizer does not contain U+2581 (ie the space) token
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 64/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Quiet
- Tech stack
- python
- Domain
- machine-learning
Research direction
Start at build_sentencepiece_luts in train_gpt.py around line 180 and inspect how normal and byte-fallback tokens contribute to the byte estimate. Reproduce the custom SentencePiece edge case described here, then add a regression test showing that val_byte_count, tokens_per_byte, and BPB are no longer distorted at word boundaries.
Written by the indexing model from the issue text.
Description
build_sentencepiece_luts (train_gpt.py:180) tries to estimate how many UTF-8 bytes each token corresponds to.
For normal tokens, it removes the leading ▁, treats that as a single space byte, and then counts the UTF-8 bytes in the rest of the token string. For byte-fallback tokens (sp.is_byte()), it just counts 1 byte per token.
The issue shows up when ▁ (U+2581) is not present as its own token. In that case, SentencePiece encodes it as three separate byte tokens: <0xE2>, <0x96>, and <0x81>, since ▁ is three bytes in UTF-8. Because each of those fallback tokens is counted as 1 byte, the code ends up treating ▁ as 3 bytes.
But in practice, ▁ is only used to represent a single ASCII space (0x20), which is just 1 byte. So every word boundary gets overcounted by 2 bytes.
That makes val_byte_count too large, which then makes tokens_per_byte look smaller than it really is and pushes BPB artificially low. I have run a few tests, we are talking roughly about a 20% decrease in reported bpb compared to the true one.
The standard fineweb_1024_bpe.model does include ▁ as token 939, so the normal stripping logic handles it correctly and this edge case never happens. It only appears with a custom SentencePiece model that does not contain a standalone ▁ token.
Hope this helps.
Riccardo
- Dominant language
- Python
- Stars
- 5.2k
- Forks
- 3.3k
- PR merge metrics
- No merged PRs in 30d
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from openai/parameter-golf
-
Difficulty 4/5 3-5 days Newbie friendliness 35/100
openai/parameter-golf#2167 · 1 comment · 1 reaction ·
-
Difficulty 3/5 1-2 days Newbie friendliness 20/100
openai/parameter-golf#2166 ·
-
Difficulty 4/5 3-5 days Newbie friendliness 25/100
openai/parameter-golf#2156 ·
-
Difficulty 4/5 3-5 days Newbie friendliness 25/100
openai/parameter-golf#2152 · 1 comment ·
-
Difficulty 5/5 Over a week Newbie friendliness 20/100
openai/parameter-golf#2151 · 1 comment ·
All issues in openai/parameter-golf
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100