Hacktoberfest 2026: những issue maintainer đã đánh dấu cho tháng Mười, đang mở và phù hợp người mới. Xem issue Hacktoberfest

[ET-VK][Adreno] Non-deterministic conv2d output for c_out in {168,176,184,192}; clean on Mali

Đang mở
  • Có gì không ổn?
#21,938 2 bình luận 0 reaction 1 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

@SS-JIA đang làm issue này rồi.

Từ ngày 24/8/2026.

Đánh giá

Issue này chưa được đánh giá.

Mô tả

module: vulkan
🐛 Describe the bug

On Adreno, a single Conv2d lowered to the Vulkan delegate produces a different result on almost every execution, for a specific band of output channel counts. Identical .pte, identical input, same already-encoded command buffer replayed in one process.

Probe: Conv2d(48, c_out, 3, padding=1, bias=False) on a (1, 48, 64, 64) input, 200 in-process replays per run, order rotated across runs, cool device.

c_out excess distinct outputs replays runs with any deviation
144 0 1600 0 / 8
160 0 1600 0 / 8
168 1142 1800 9 / 9
176 427 800 4 / 4
184 770 1000 5 / 5
192 772 1200 6 / 6
200 0 1400 0 / 7
224 0 600 0 / 3

Perfect separation. Every run of every c_out in {168, 176, 184, 192} deviates; no run of {144, 160, 200, 224} deviates across 5200 replays.

The band lies inside the im2col/GEMM convolution path: should_use_conv2d_im2col() routes any conv with c_out >= kIm2colMinCOut (128) there. Forcing that function to return false, so the direct conv2d shader is used, makes all of these deterministic. Note 160 and 200 are also in that path and are clean, so it is a size band inside the path, not the threshold itself.

This is very likely an Adreno driver or hardware problem, not an ExecuTorch bug

Three independent checks point away from ExecuTorch:

  1. A different GPU vendor is clean. The same binaries and the same .pte files on a Galaxy S10+ (SM-G975F, Exynos 9820, Mali-G76, Android 12): all eight c_out values, 200 replays each, zero deviations. This is the strongest signal, because should_use_conv2d_im2col() returns graph.device_is_mali() || c_out >= kIm2colMinCOut, so on Mali every conv takes the im2col path, including 144 and 160 which do not on Adreno. If the im2col shader itself were wrong, Mali should have been the worse device.
    Confirmed at the application level as well, not just with executor_runner: a demo app built from the stock Vulkan backend runs both object detection (YOLO26) and image classification (EfficientNetV2-S) on the S10 with correct and stable results.

Separately, and worth knowing for anyone weighing the Vulkan delegate on Mali: on this S10 the Vulkan delegate is close to 2x slower than the XNNPACK equivalent for the same models. That is the opposite of the Adreno device, where Vulkan is clearly faster. The S10 is a 2019 part so this is not necessarily representative of current Mali hardware, but it does mean the delegate is fast on the GPU where it is incorrect and slow on the GPU where it is correct.

  1. Output is numerically correct on Mali, not merely stable. Exporting a Vulkan .pte and a CPU .pte from the same model instance and comparing both against the eager reference gives 9.9e-07 (Vulkan) and 3.6e-07 (CPU), i.e. fp32 rounding.
  2. Khronos synchronization validation reports zero hazards on the failing workload (layer injected into a debuggable app via gpu_debug_layers, sync validation confirmed active by the layer's own settings message). No SYNC-HAZARD, no VUID errors.

So the return false workaround most likely helps by avoiding whatever the Adreno driver miscompiles, rather than by avoiding an incorrect shader.

Reproducing

Counting distinct md5 of executor_runner stdout across repeated process launches is a poor measure: it is noisy, and on this device the failure rate of the related issue #21939 depends strongly on GPU power state. Patch executor_runner to dump outputs on every iteration of --num_executions (stock writes only the last), then count distinct outputs across in-process replays. This one is stable and reproduces at 215 to 262 MHz on a cool device, so unlike #21939 it is not sensitive to clock.

Possibly the same underlying defect as #21939, which is also Adreno-only but low-rate and power-state dependent.

Versions

ExecuTorch 1.4.1. Failing: Galaxy S26 Ultra (SM-S948B), Adreno 840v2. Clean: Galaxy S10+ (SM-G975F), Mali-G76.

Ngôn ngữ chính
Python
Star
5.1k
Fork
1.2k
Merge trung bình
1 ngày 23 giờ
Pull request đã merge (30 ngày)
713

Chuẩn bị môi trường

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của pytorch/executorch

Tất cả issue của pytorch/executorch

Issue tương tự

Thêm issue về Python

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.