Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

[muse-glimmer] interpolate_pos_emb returns corrupted bf16 values on CPU and can segfault

Abierto
  • ¿Algo no está bien?
#23,797 2 comentarios 0 reacciones 1 asignado Ver en GitHub

Los mantenedores suelen responder en 1 día

@JakeStevens ya está trabajando en esto.

Desde el 14/10/2026.

  • #23798 de @AxelNoun — abierto

Evaluación

Este issue todavía no se ha evaluado.

Descripción

bug module: examples triaged
🐛 Describe the bug

interpolate_pos_emb in examples/models/muse-glimmer/vision/precompute.py returns wrong values for a bf16 positional table on CPU, and it is the likely source of the intermittent segfault in the pull / unittest / macos / macos-job job.

In CI. The job crashes with Fatal Python error: Segmentation fault in torch.nn.functional.grid_sample, called from interpolate_pos_emb, while running examples/models/muse-glimmer/tests/test_vision_precompute.py. I found two occurrences, both with torch 2.15.0a0+gite91899c on osx-arm64: on main at 69eeed2 (run 38405593574, in PrecomputeBundleTest::test_all_inputs_consistent) and on #22373 (run 38325606287, in PosEmbInterpTest::test_downscaled_shape). The crashing thread in the second one:

Fatal Python error: Segmentation fault
Current thread 0x00000001f0e7a140 (most recent call first):
  File ".../site-packages/torch/nn/functional.py", line 5701 in grid_sample
  File ".../site-packages/executorch/examples/models/muse_glimmer/vision/precompute.py", line 114 in interpolate_pos_emb
  File ".../examples/models/muse-glimmer/tests/test_vision_precompute.py", line 72 in test_downscaled_shape

Values. The output is wrong on every run, not only when the process crashes. Interpolating to the native 32x32 grid puts every sample on a texel center, so it should return the table exactly:

import torch
from executorch.examples.models.muse_glimmer.vision.precompute import (
    MuseGlimmerVisionConfig,
    interpolate_pos_emb,
)

torch.manual_seed(0)
cfg = MuseGlimmerVisionConfig()
table = torch.randn(cfg.pos_emb_grid_h * cfg.pos_emb_grid_w, cfg.latent_dim, dtype=torch.bfloat16)
pos = interpolate_pos_emb(table, 32, 32, cfg)  # native 32x32 grid: should return the table
print("mismatched:", (pos[0] != table).float().mean().item(), "NaN:", int(pos.isnan().sum()))

mismatched: 0.9780623316764832 NaN: 826

Cause. interpolate_pos_emb passes the permuted view pos_embed_table.view(32, 32, 1536).permute(2, 0, 1).unsqueeze(0), whose strides are (1536, 1, 49152, 1536), to F.grid_sample. For Half and BFloat16, the vectorized CPU kernel computes its gather offsets in int_same_size_t<scalar_t>, that is int16_t (GridSamplerKernel.cpp, lines 468 to 469, offsets at lines 568 to 571), and grid_sampler_2d_cpu guards against offset overflow only for float32 (GridSampler.cpp, line 1101). For this view the offsets go up to 31 * 49152 + 31 * 1536 = 1571328, far above the int16 maximum of 32767, so they wrap. An emulation of that int16 arithmetic puts about half of the gathered reads before the start of the storage: they return garbage or NaN, and can fault when that memory is unmapped, which is consistent with the CI crash. On the contiguous tensor, the same 32x32 call returns the table exactly, and an 8x8 downscale matches a float32 reference within bf16 rounding. The kernel code is the same on PyTorch main (afa81a4) and at the CI pin (e91899c); the 16-bit dtypes reached this kernel with pytorch/pytorch#134812, first released in 2.6.0.

Impact. precompute_vision_inputs is called by inference.py (generate_with_image, whose vision encoder adds pos_emb to the patch embeddings) and by export/common.py (sample_vision_inputs, example inputs for export). The C++ runtime does not go through this code: vision/preprocess.h interpolates from the float32 table. The existing tests check only shapes and dtypes, which is why the wrong values went unnoticed.

Proposed fix. Make the view contiguous before grid_sample, which keeps the computation in the table's dtype and brings the largest gather offset to 1023, and add value checks to the tests. I have a PR ready with this change.

A separate observation, not addressed by that fix: the Python code computes the sampling coordinates in bf16 (it builds the grid in bf16, and grid_sample, which requires the grid in the input dtype, also unnormalizes it in bf16), while vision/preprocess.h computes them in double. On grids that are not powers of two the results then diverge: on 74x74, a Python transcription of detail::interpolate_pos_emb (I did not build the C++) differs from the corrected Python output on about 97% of the values, by up to about 0.75. The docstring says the interpolation stays in the table's dtype to preserve reference numerics, so this may be intended. Which of the two paths is meant to match the reference?

Versions

Local reproduction: Windows 11, x86_64 (AVX2), Python 3.12.11, torch 2.14.0+cpu, executorch 1.5.1 (its precompute.py is identical to the one on main at 060075b). A pure PyTorch transcription of the reproduction also fails with torch 2.11.0+cu128 on Python 3.14.2 (97.5% of the values mismatched, 826 to 842 NaN depending on the run).

CI: osx-arm64, torch 2.15.0a0+gite91899c.

cc @mergennachin @iseeyuan @lucylq @helunwencser @tarun292 @kimishpatel @jackzhxng

Lenguaje dominante
Python
Estrellas
5.1k
Forks
1.2k
Merge medio
2 d 48 min
PR fusionados (30 d)
693

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de pytorch/executorch

Todos los issues de pytorch/executorch

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.