MPS: GPT-2 produces NaN logits and invalid token IDs
Maintainers usually reply within 1 day
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 70/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Active
- Tech stack
- cpp, objective-c
- Domain
- backend, machine-learning
Research direction
Start in src/mps/kernels.mm: the two tanh calls backing GELU (and the other activation kernel) are the reported source of NaN on MPS, and the issue already says precise::tanh fixes the tested FP32/FP16/BF16 cases. Read the surrounding kernel definitions and the MPS build setup first, then rebuild on Apple silicon and run the conversion-plus-device="mps" script from the issue. Done means generate_batch returns valid token IDs with finite logits matching the CPU run for the FP32, FP16 and BF16 cases.
Written by the indexing model from the issue text.
Description
Reproduced with CTranslate2 4.9.0, Apple M4 Pro, macOS 15.7.7.
import ctranslate2 as ct
from transformers import AutoTokenizer
name = "openai-community/gpt2"
ct.converters.TransformersConverter(name).convert(
"gpt2-ct2", quantization="float32"
)
tokenizer = AutoTokenizer.from_pretrained(name)
model = ct.Generator("gpt2-ct2", device="mps", compute_type="float32")
prompt = tokenizer.tokenize("The capital of France is")
model.generate_batch([prompt], beam_size=2, max_length=16)
Actual: Raises:
Invalid token ID 18446744073709551615:
valid IDs are between 0 and 50256
Greedy decoding returns !!!!!!!!!!!!!!!! with score -inf. All 251,285 forward logits are nonfinite. The same converted model works on CPU.
Cause identified: MPS returns NaN for finite inputs such as tanh-based GELU(20) and tanh(100). GPT-2 first produces NaN in its first feed-forward layer.
Replacing the two tanh calls in src/mps/kernels.mm with precise::tanh fixes the tested FP32, FP16 and BF16 cases in an isolated build.
Separate from #2153.
- Dominant language
- C++
- Stars
- 4.8k
- Forks
- 557
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 24
Getting set up
- No Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from OpenNMT/CTranslate2
-
Allow no-timestamp Whisper decoding beyond 224 tokensPossibly taken @Emre-Akgul claimed this 127 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
OpenNMT/CTranslate2#2074 ·
Maintainers usually reply within 1 day
-
T5Gemma fails on MPS: incorrect beam expansion with padded batchesPossibly taken @TBO22 claimed this today. Open
Difficulty 4/5 3-5 days Newbie friendliness 30/100
OpenNMT/CTranslate2#2153 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 5/5 Over a week Newbie friendliness 8/100
OpenNMT/CTranslate2#2152 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 3/5 1-2 days Newbie friendliness 55/100
OpenNMT/CTranslate2#2149 ·
Maintainers usually reply within 1 day
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
OpenNMT/CTranslate2#2105 ·
Maintainers usually reply within 1 day
All issues in OpenNMT/CTranslate2
Similar issues
-
Documentation: Fix parameter markup, legacy branch link, and phrasing in Detachable Joints and Model Command tutorialsPossibly taken @patelshivani2283-lab claimed this today. Open
Difficulty 1/5 1-3 hours Newbie friendliness 88/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
Maintainers usually reply within 1 day
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
Niko1221/Strata#6071 · 2 comments ·
Maintainers usually reply within 4 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
KhronosGroup/Vulkan-Tutorial#551 ·
Maintainers usually reply within 1 day