Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

[ET-VK][Adreno] Non-deterministic conv2d output for c_out in {168,176,184,192}; clean on Mali

Abierto
  • ¿Algo no está bien?
#21,938 2 comentarios 0 reacciones 1 asignado Ver en GitHub

Los mantenedores suelen responder en 1 día

@SS-JIA ya está trabajando en esto.

Desde el 24/8/2026.

Evaluación

Este issue todavía no se ha evaluado.

Descripción

module: vulkan
🐛 Describe the bug

On Adreno, a single Conv2d lowered to the Vulkan delegate produces a different result on almost every execution, for a specific band of output channel counts. Identical .pte, identical input, same already-encoded command buffer replayed in one process.

Probe: Conv2d(48, c_out, 3, padding=1, bias=False) on a (1, 48, 64, 64) input, 200 in-process replays per run, order rotated across runs, cool device.

c_out excess distinct outputs replays runs with any deviation
144 0 1600 0 / 8
160 0 1600 0 / 8
168 1142 1800 9 / 9
176 427 800 4 / 4
184 770 1000 5 / 5
192 772 1200 6 / 6
200 0 1400 0 / 7
224 0 600 0 / 3

Perfect separation. Every run of every c_out in {168, 176, 184, 192} deviates; no run of {144, 160, 200, 224} deviates across 5200 replays.

The band lies inside the im2col/GEMM convolution path: should_use_conv2d_im2col() routes any conv with c_out >= kIm2colMinCOut (128) there. Forcing that function to return false, so the direct conv2d shader is used, makes all of these deterministic. Note 160 and 200 are also in that path and are clean, so it is a size band inside the path, not the threshold itself.

This is very likely an Adreno driver or hardware problem, not an ExecuTorch bug

Three independent checks point away from ExecuTorch:

  1. A different GPU vendor is clean. The same binaries and the same .pte files on a Galaxy S10+ (SM-G975F, Exynos 9820, Mali-G76, Android 12): all eight c_out values, 200 replays each, zero deviations. This is the strongest signal, because should_use_conv2d_im2col() returns graph.device_is_mali() || c_out >= kIm2colMinCOut, so on Mali every conv takes the im2col path, including 144 and 160 which do not on Adreno. If the im2col shader itself were wrong, Mali should have been the worse device.
    Confirmed at the application level as well, not just with executor_runner: a demo app built from the stock Vulkan backend runs both object detection (YOLO26) and image classification (EfficientNetV2-S) on the S10 with correct and stable results.

Separately, and worth knowing for anyone weighing the Vulkan delegate on Mali: on this S10 the Vulkan delegate is close to 2x slower than the XNNPACK equivalent for the same models. That is the opposite of the Adreno device, where Vulkan is clearly faster. The S10 is a 2019 part so this is not necessarily representative of current Mali hardware, but it does mean the delegate is fast on the GPU where it is incorrect and slow on the GPU where it is correct.

  1. Output is numerically correct on Mali, not merely stable. Exporting a Vulkan .pte and a CPU .pte from the same model instance and comparing both against the eager reference gives 9.9e-07 (Vulkan) and 3.6e-07 (CPU), i.e. fp32 rounding.
  2. Khronos synchronization validation reports zero hazards on the failing workload (layer injected into a debuggable app via gpu_debug_layers, sync validation confirmed active by the layer's own settings message). No SYNC-HAZARD, no VUID errors.

So the return false workaround most likely helps by avoiding whatever the Adreno driver miscompiles, rather than by avoiding an incorrect shader.

Reproducing

Counting distinct md5 of executor_runner stdout across repeated process launches is a poor measure: it is noisy, and on this device the failure rate of the related issue #21939 depends strongly on GPU power state. Patch executor_runner to dump outputs on every iteration of --num_executions (stock writes only the last), then count distinct outputs across in-process replays. This one is stable and reproduces at 215 to 262 MHz on a cool device, so unlike #21939 it is not sensitive to clock.

Possibly the same underlying defect as #21939, which is also Adreno-only but low-rate and power-state dependent.

Versions

ExecuTorch 1.4.1. Failing: Galaxy S26 Ultra (SM-S948B), Adreno 840v2. Clean: Galaxy S10+ (SM-G975F), Mali-G76.

Lenguaje dominante
Python
Estrellas
5.1k
Forks
1.2k
Merge medio
2 d 10 h
PR fusionados (30 d)
632

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de pytorch/executorch

Todos los issues de pytorch/executorch

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.