Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

[ET-VK][Adreno] Low-rate non-deterministic CNN output, dependent on GPU power state; clean on Mali

Abierto
  • ¿Algo no está bien?
#21,939 1 comentario 0 reacciones 1 asignado Ver en GitHub

Los mantenedores suelen responder en 1 día

@SS-JIA ya está trabajando en esto.

Desde el 24/8/2026.

Evaluación

Este issue todavía no se ha evaluado.

Descripción

module: vulkan
🐛 Describe the bug

On Adreno, CNNs lowered to the Vulkan delegate produce different results run to run, at a low but persistent rate, even with the im2col/GEMM path from #21938 disabled. EfficientNetV2-S gives 24 distinct results in 25 replays of the same already-encoded command buffer inside a single process, which rules out load-time state, prepack and weight packing.

Deviation magnitude: EfficientNetV2-S logits span [-1.44, 2.75]; typical deviations are 0.001 to 0.009, with rare excursions to about 4.0. argmax was stable across 30 runs for one input, so class flips need the rare large excursions.

This is very likely an Adreno driver or hardware problem, not an ExecuTorch bug
  1. A different GPU vendor is clean. Same binaries, same .pte, Galaxy S10+ (Exynos 9820, Mali-G76): 20 runs, 4020 in-process replays, zero deviations. The probe below that deviates on Adreno gives 0 across 1200 replays there. Output is also numerically correct on Mali, not merely stable: matched Vulkan and CPU exports from one model instance both land within fp32 rounding of the eager reference (9.9e-07 and 3.6e-07).
    Confirmed at the application level as well, not just with executor_runner: a demo app built from the stock Vulkan backend runs both object detection (YOLO26) and image classification (EfficientNetV2-S) on the S10 with correct and stable results.

Separately, and worth knowing for anyone weighing the Vulkan delegate on Mali: on this S10 the Vulkan delegate is close to 2x slower than the XNNPACK equivalent for the same models. That is the opposite of the Adreno device, where Vulkan is clearly faster. The S10 is a 2019 part so this is not necessarily representative of current Mali hardware, but it does mean the delegate is fast on the GPU where it is incorrect and slow on the GPU where it is correct.

  1. Khronos synchronization validation reports zero hazards on the failing workload. Layer injected into a debuggable app via gpu_debug_layers, sync validation confirmed active by the layer's own settings message. No SYNC-HAZARD, no VUID errors of any kind.
  2. Unconditional full memory barriers change nothing. Patching CommandBuffer::insert_barrier to always emit a global ALL_COMMANDS -> ALL_COMMANDS barrier with VK_ACCESS_MEMORY_READ_BIT | VK_ACCESS_MEMORY_WRITE_BIT before every dispatch, on top of the computed per-resource barriers, does not help.

The dispatch sequence is also deterministic: an env-gated trace in DispatchNode::encode gives byte-identical traces across runs.

Measurement note, important for anyone reproducing

The failure rate is strongly dependent on GPU power state and swings by more than an order of magnitude for the same .pte and same binary within minutes. Same build, EfficientNetV2-S: 79 distinct / 101 at ~1000 MHz, 13 / 101 at ~826 MHz, 1 / 101 pinned at 461 MHz.

adb shell cmd power set-fixed-performance-mode-enabled true pins the GPU at 461 MHz without root on this device (devfreq/{min,max}_freq are unavailable). Over 4 alternating pairs of 100 in-process replays: 3 excess distinct / 404 pinned versus 30 / 404 unpinned.

Consequences for methodology:

  • Any A/B comparison that is not interleaved within a single round is unreliable.
  • Patch executor_runner to dump outputs on every iteration of --num_executions; stock writes only the last. This is far more sensitive than relaunching the process.
  • Probe models must be activation calibrated. Untrained stacked convs collapse the signal to ~1e-19, where "distinct output" counts last-bit noise on values carrying no information.
Smallest structure that shows it

12 rounds, order rotated each round, 200 in-process replays per run, calibrated probes, input (1, 48, 64, 64), im2col disabled. Excess distinct outputs per 2400 replays:

graph run A run B
conv3x3 -> SiLU -> conv1x1 43 18
conv3x3 -> ReLU -> conv1x1 3 11
conv3x3 -> Sigmoid -> conv1x1 - 2
conv3x3 -> conv1x1 1 -
conv3x3 -> SiLU -> conv3x3 0 0

conv -> SiLU -> conv3x3 is 0 across 4800 replays while the same graph with a 1x1 consumer is the worst case. With no activation at all, kernel size alone does nothing (0 to 6 per 2400 for all four combinations of 1x1/3x3 producer and consumer). Absolute rates are low, so treat this as a lead rather than a conclusion. It is only visible with DVFS active; pinned at 461 MHz all four collapse to the noise floor.

Forcing 1x1 convs away from Conv2dMethod::Pointwise into the generic sliding window shader does not fix it, so it is not shader identity. The same generic shader is clean at 3x3 and dirty at 1x1.

Ruled out, each tested

Memory aliasing (add_tensor(..., -1) instead of mem_obj_id(); made it worse), storage type, command buffer splitting (execute_threshold_node_count set to SIZE_MAX), descriptor pool recycling (descriptor_pool_safety_factor 1.25 to 64.0), caller threading, host cache coherency, node count.

Possibly the same underlying defect as #21938, which is also Adreno-only but is 100% reproducible and independent of clock.

Versions

ExecuTorch 1.4.1. Failing: Galaxy S26 Ultra (SM-S948B), Adreno 840v2. Clean: Galaxy S10+ (SM-G975F), Mali-G76.

Lenguaje dominante
Python
Estrellas
5.1k
Forks
1.2k
Merge medio
2 d 9 h
PR fusionados (30 d)
632

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de pytorch/executorch

Todos los issues de pytorch/executorch

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.