Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

[muse-glimmer] interpolate_pos_emb returns corrupted bf16 values on CPU and can segfault

Aperta
  • Qualcosa non va?
#23,797 2 commenti 0 reazioni 1 assegnatario Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

@JakeStevens ci sta già lavorando.

Dal 14/10/2026.

  • #23798 di @AxelNoun — aperta

Valutazione

Questa issue non è ancora stata valutata.

Descrizione

bug module: examples triaged
🐛 Describe the bug

interpolate_pos_emb in examples/models/muse-glimmer/vision/precompute.py returns wrong values for a bf16 positional table on CPU, and it is the likely source of the intermittent segfault in the pull / unittest / macos / macos-job job.

In CI. The job crashes with Fatal Python error: Segmentation fault in torch.nn.functional.grid_sample, called from interpolate_pos_emb, while running examples/models/muse-glimmer/tests/test_vision_precompute.py. I found two occurrences, both with torch 2.15.0a0+gite91899c on osx-arm64: on main at 69eeed2 (run 38405593574, in PrecomputeBundleTest::test_all_inputs_consistent) and on #22373 (run 38325606287, in PosEmbInterpTest::test_downscaled_shape). The crashing thread in the second one:

Fatal Python error: Segmentation fault
Current thread 0x00000001f0e7a140 (most recent call first):
  File ".../site-packages/torch/nn/functional.py", line 5701 in grid_sample
  File ".../site-packages/executorch/examples/models/muse_glimmer/vision/precompute.py", line 114 in interpolate_pos_emb
  File ".../examples/models/muse-glimmer/tests/test_vision_precompute.py", line 72 in test_downscaled_shape

Values. The output is wrong on every run, not only when the process crashes. Interpolating to the native 32x32 grid puts every sample on a texel center, so it should return the table exactly:

import torch
from executorch.examples.models.muse_glimmer.vision.precompute import (
    MuseGlimmerVisionConfig,
    interpolate_pos_emb,
)

torch.manual_seed(0)
cfg = MuseGlimmerVisionConfig()
table = torch.randn(cfg.pos_emb_grid_h * cfg.pos_emb_grid_w, cfg.latent_dim, dtype=torch.bfloat16)
pos = interpolate_pos_emb(table, 32, 32, cfg)  # native 32x32 grid: should return the table
print("mismatched:", (pos[0] != table).float().mean().item(), "NaN:", int(pos.isnan().sum()))

mismatched: 0.9780623316764832 NaN: 826

Cause. interpolate_pos_emb passes the permuted view pos_embed_table.view(32, 32, 1536).permute(2, 0, 1).unsqueeze(0), whose strides are (1536, 1, 49152, 1536), to F.grid_sample. For Half and BFloat16, the vectorized CPU kernel computes its gather offsets in int_same_size_t<scalar_t>, that is int16_t (GridSamplerKernel.cpp, lines 468 to 469, offsets at lines 568 to 571), and grid_sampler_2d_cpu guards against offset overflow only for float32 (GridSampler.cpp, line 1101). For this view the offsets go up to 31 * 49152 + 31 * 1536 = 1571328, far above the int16 maximum of 32767, so they wrap. An emulation of that int16 arithmetic puts about half of the gathered reads before the start of the storage: they return garbage or NaN, and can fault when that memory is unmapped, which is consistent with the CI crash. On the contiguous tensor, the same 32x32 call returns the table exactly, and an 8x8 downscale matches a float32 reference within bf16 rounding. The kernel code is the same on PyTorch main (afa81a4) and at the CI pin (e91899c); the 16-bit dtypes reached this kernel with pytorch/pytorch#134812, first released in 2.6.0.

Impact. precompute_vision_inputs is called by inference.py (generate_with_image, whose vision encoder adds pos_emb to the patch embeddings) and by export/common.py (sample_vision_inputs, example inputs for export). The C++ runtime does not go through this code: vision/preprocess.h interpolates from the float32 table. The existing tests check only shapes and dtypes, which is why the wrong values went unnoticed.

Proposed fix. Make the view contiguous before grid_sample, which keeps the computation in the table's dtype and brings the largest gather offset to 1023, and add value checks to the tests. I have a PR ready with this change.

A separate observation, not addressed by that fix: the Python code computes the sampling coordinates in bf16 (it builds the grid in bf16, and grid_sample, which requires the grid in the input dtype, also unnormalizes it in bf16), while vision/preprocess.h computes them in double. On grids that are not powers of two the results then diverge: on 74x74, a Python transcription of detail::interpolate_pos_emb (I did not build the C++) differs from the corrected Python output on about 97% of the values, by up to about 0.75. The docstring says the interpolation stays in the table's dtype to preserve reference numerics, so this may be intended. Which of the two paths is meant to match the reference?

Versions

Local reproduction: Windows 11, x86_64 (AVX2), Python 3.12.11, torch 2.14.0+cpu, executorch 1.5.1 (its precompute.py is identical to the one on main at 060075b). A pure PyTorch transcription of the reproduction also fails with torch 2.11.0+cu128 on Python 3.14.2 (97.5% of the values mismatched, 826 to 842 NaN depending on the run).

CI: osx-arm64, torch 2.15.0a0+gite91899c.

cc @mergennachin @iseeyuan @lucylq @helunwencser @tarun292 @kimishpatel @jackzhxng

Lingua principale
Python
Stelle
5.1k
Fork
1.2k
Merge medio
2g 51m
PR unite (30g)
712

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di pytorch/executorch

Tutte le issue di pytorch/executorch

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.