Hacktoberfest 2026：维护者为十月标记出来的 issue，仍然开放、适合新手。 浏览 Hacktoberfest issue

[ET-VK][Adreno] Non-deterministic conv2d output for c_out in {168,176,184,192}; clean on Mali

未关闭
  • 哪里不对？
#21,938 2 条评论 0 个 reaction 已指派 1 人 在 GitHub 查看

维护者通常 1 天内回复

@SS-JIA 已经在做这个了。

开始于 2026年8月24日。

评估

这个 Issue 还没有评估数据。

描述

module: vulkan
🐛 Describe the bug

On Adreno, a single Conv2d lowered to the Vulkan delegate produces a different result on almost every execution, for a specific band of output channel counts. Identical .pte, identical input, same already-encoded command buffer replayed in one process.

Probe: Conv2d(48, c_out, 3, padding=1, bias=False) on a (1, 48, 64, 64) input, 200 in-process replays per run, order rotated across runs, cool device.

c_out excess distinct outputs replays runs with any deviation
144 0 1600 0 / 8
160 0 1600 0 / 8
168 1142 1800 9 / 9
176 427 800 4 / 4
184 770 1000 5 / 5
192 772 1200 6 / 6
200 0 1400 0 / 7
224 0 600 0 / 3

Perfect separation. Every run of every c_out in {168, 176, 184, 192} deviates; no run of {144, 160, 200, 224} deviates across 5200 replays.

The band lies inside the im2col/GEMM convolution path: should_use_conv2d_im2col() routes any conv with c_out >= kIm2colMinCOut (128) there. Forcing that function to return false, so the direct conv2d shader is used, makes all of these deterministic. Note 160 and 200 are also in that path and are clean, so it is a size band inside the path, not the threshold itself.

This is very likely an Adreno driver or hardware problem, not an ExecuTorch bug

Three independent checks point away from ExecuTorch:

  1. A different GPU vendor is clean. The same binaries and the same .pte files on a Galaxy S10+ (SM-G975F, Exynos 9820, Mali-G76, Android 12): all eight c_out values, 200 replays each, zero deviations. This is the strongest signal, because should_use_conv2d_im2col() returns graph.device_is_mali() || c_out >= kIm2colMinCOut, so on Mali every conv takes the im2col path, including 144 and 160 which do not on Adreno. If the im2col shader itself were wrong, Mali should have been the worse device.
    Confirmed at the application level as well, not just with executor_runner: a demo app built from the stock Vulkan backend runs both object detection (YOLO26) and image classification (EfficientNetV2-S) on the S10 with correct and stable results.

Separately, and worth knowing for anyone weighing the Vulkan delegate on Mali: on this S10 the Vulkan delegate is close to 2x slower than the XNNPACK equivalent for the same models. That is the opposite of the Adreno device, where Vulkan is clearly faster. The S10 is a 2019 part so this is not necessarily representative of current Mali hardware, but it does mean the delegate is fast on the GPU where it is incorrect and slow on the GPU where it is correct.

  1. Output is numerically correct on Mali, not merely stable. Exporting a Vulkan .pte and a CPU .pte from the same model instance and comparing both against the eager reference gives 9.9e-07 (Vulkan) and 3.6e-07 (CPU), i.e. fp32 rounding.
  2. Khronos synchronization validation reports zero hazards on the failing workload (layer injected into a debuggable app via gpu_debug_layers, sync validation confirmed active by the layer's own settings message). No SYNC-HAZARD, no VUID errors.

So the return false workaround most likely helps by avoiding whatever the Adreno driver miscompiles, rather than by avoiding an incorrect shader.

Reproducing

Counting distinct md5 of executor_runner stdout across repeated process launches is a poor measure: it is noisy, and on this device the failure rate of the related issue #21939 depends strongly on GPU power state. Patch executor_runner to dump outputs on every iteration of --num_executions (stock writes only the last), then count distinct outputs across in-process replays. This one is stable and reproduces at 215 to 262 MHz on a cool device, so unlike #21939 it is not sensitive to clock.

Possibly the same underlying defect as #21939, which is also Adreno-only but low-rate and power-state dependent.

Versions

ExecuTorch 1.4.1. Failing: Galaxy S26 Ultra (SM-S948B), Adreno 840v2. Clean: Galaxy S10+ (SM-G975F), Mali-G76.

主要语言
Python
星标
5.1k
派生
1.2k
平均合并
2 天 10 小时
30 天内合并 PR
632

环境准备

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

pytorch/executorch 的其他 Issue

查看 pytorch/executorch 的全部 Issue

相似的 Issue

更多 Python Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。