MPS: GPT-2 produces NaN logits and invalid token IDs

Open Beginner friendly
#2,154 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

@TBO22 is already working on this.

Since Nov 15, 2026.

  • #2155 by @TBO22 — open

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
70/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
cpp, objective-c

Research direction

Start in src/mps/kernels.mm: the two tanh calls backing GELU (and the other activation kernel) are the reported source of NaN on MPS, and the issue already says precise::tanh fixes the tested FP32/FP16/BF16 cases. Read the surrounding kernel definitions and the MPS build setup first, then rebuild on Apple silicon and run the conversion-plus-device="mps" script from the issue. Done means generate_batch returns valid token IDs with finite logits matching the CPU run for the FP32, FP16 and BF16 cases.

Written by the indexing model from the issue text.

Description

Reproduced with CTranslate2 4.9.0, Apple M4 Pro, macOS 15.7.7.

import ctranslate2 as ct
from transformers import AutoTokenizer

name = "openai-community/gpt2"
ct.converters.TransformersConverter(name).convert(
    "gpt2-ct2", quantization="float32"
)
tokenizer = AutoTokenizer.from_pretrained(name)
model = ct.Generator("gpt2-ct2", device="mps", compute_type="float32")
prompt = tokenizer.tokenize("The capital of France is")

model.generate_batch([prompt], beam_size=2, max_length=16)

Actual: Raises:

Invalid token ID 18446744073709551615:
valid IDs are between 0 and 50256

Greedy decoding returns !!!!!!!!!!!!!!!! with score -inf. All 251,285 forward logits are nonfinite. The same converted model works on CPU.

Cause identified: MPS returns NaN for finite inputs such as tanh-based GELU(20) and tanh(100). GPT-2 first produces NaN in its first feed-forward layer.

Replacing the two tanh calls in src/mps/kernels.mm with precise::tanh fixes the tested FP32, FP16 and BF16 cases in an isolated build.

Separate from #2153.

Dominant language
C++
Stars
4.8k
Forks
557
Avg merge
1d 6h
Merged PRs (30d)
24

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from OpenNMT/CTranslate2

All issues in OpenNMT/CTranslate2

Similar issues

More C++ issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.