T5Gemma fails on MPS: incorrect beam expansion with padded batches

Open
#2,153 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

@TBO22 is already working on this.

Since Nov 15, 2026.

  • #2155 by @TBO22 — open

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
30/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
cpp, python

Research direction

Start from MultiHeadAttention::forward_merged, where the payload says beam size is inferred from memory->dim(0) after padding removal; trace how the padded batch is restored before encoder keys/values are expanded for beam search. Reproduce with the given script on an MPS device to confirm the LLDB shapes (decoder [16, 4, 1, 256] vs encoder [8, 4, 348, 256]) at the concatenation. Done when beam size comes from the restored encoder key tensor's batch dimension and a regression test covers a variable-length batch with beam_size=2.

Written by the indexing model from the issue text.

Description

T5Gemma generation fails with PyPI CTranslate2 4.9.0:

RuntimeError: MPS pointer is not backed by a registered Metal buffer

Observed in CI and reproduced on an M4 Pro running macOS 15.7.7.

Reproduction
pip install ctranslate2==4.9.0 transformers==5.22.2 \
  torch==2.15.0 datasets==5.1.0 sentencepiece==0.2.3
from pathlib import Path

import ctranslate2
from datasets import load_dataset
from transformers import AutoTokenizer

model = "jordimas/t5gemma-2-270m-270m"
revision = "dfb6d0d619d8e6d42f279a6197abb90dec0137c1"
output_dir = "t5gemma-ct2-float32"

if not Path(output_dir).exists():
    ctranslate2.converters.TransformersConverter(
        model, revision=revision
    ).convert(output_dir, quantization="float32")

tokenizer = AutoTokenizer.from_pretrained(model, revision=revision)
samples = load_dataset(
    "rajpurkar/squad",
    split="validation",
    revision="7b6d24c440a36b6815f21b70d25016731768db1f",
).shuffle(seed=42).select(range(8))

inputs = [
    tokenizer.convert_ids_to_tokens(tokenizer.encode(
        f"Context: {s['context']} Question: {s['question']} Answer:",
        truncation=True, max_length=512,
    ))
    for s in samples
]

translator = ctranslate2.Translator(
    output_dir, device="mps", compute_type="float32",
    intra_threads=2, inter_threads=1,
)
translator.translate_batch(
    inputs, beam_size=2, sampling_topk=1,
    max_decoding_length=64, max_batch_size=8,
)
Cause and suggested fix

On the first decoding step, MultiHeadAttention::forward_merged infers beam size from memory->dim(0). After padding removal, this dimension counts packed tokens rather than examples, so encoder keys/values are not expanded for beam search.

LLDB confirmed mismatched shapes at concatenation:

  • Decoder keys: [16, 4, 1, 256]
  • Encoder keys: [8, 4, 348, 256]

Metal concatenation consequently requests more bytes than the encoder buffer contains.

Infer beam size from the restored encoder key tensor's batch dimension before expansion. Add regression coverage for variable-length batches with beam_size=2.

Dominant language
C++
Stars
4.8k
Forks
557
Avg merge
1d 6h
Merged PRs (30d)
24

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from OpenNMT/CTranslate2

All issues in OpenNMT/CTranslate2

Similar issues

More C++ issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.