Perplexity metric fails with GPT-2 tokenizer
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 74/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Quiet
- Tech stack
- python, pytorch
- Domain
- machine-learning
Research direction
Start by running the GPT-2 reproduction from the issue and inspect metrics/perplexity/perplexity.py, especially the special-token lookup and tokenizer call. Confirm that perplexity.compute works with model_id="gpt2" without either reported error; the issue does not name a regression test, so document or add coverage according to the repository’s existing metric-test conventions.
Written by the indexing model from the issue text.
Description
Description
While investigating the perplexity metric, I found that it currently fails when used with GPT-2.
Reproduction
import evaluate
perplexity = evaluate.load("perplexity", module_type="metric")
perplexity.compute(
predictions=["Hello world."],
model_id="gpt2",
)
Error 1
On my setup, the metric fails with:
AttributeError: GPT2Tokenizer has no attribute special_tokens_map_extended
The error originates from:
existing_special_tokens = list(tokenizer.special_tokens_map_extended.values())
in metrics/perplexity/perplexity.py.
Error 2
After bypassing that code path, the metric fails with:
ValueError: Asking to pad but the tokenizer does not have a padding token.
The tokenizer is called with:
padding=True
but GPT-2 does not define a padding token by default.
Environment
- evaluate: 0.4.7.dev0
- transformers: 5.12.1
- datasets: 5.0.0
- torch: 2.12.0+cu130
Notes
GPT-2 is used throughout the perplexity metric examples and documentation, so I would expect the metric to work with GPT-2 out of the box.
I'd be happy to investigate further and submit a PR with a fix.
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 343
- PR merge metrics
- No merged PRs in 30d
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from huggingface/evaluate
-
Difficulty 1/5 Under an hour Newbie friendliness 62/100
huggingface/evaluate#679 · 1 reaction ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
huggingface/evaluate#669 · 3 comments · 7 reactions ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
huggingface/evaluate#547 · 2 comments ·
-
Difficulty 1/5 Under an hour Newbie friendliness 78/100
huggingface/evaluate#545 · 2 comments ·
-
Difficulty 3/5 1-2 days Newbie friendliness 68/100
huggingface/evaluate#803 ·
All issues in huggingface/evaluate
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100