Hacktoberfest 2026：メンテナが10月に向けて印を付けた、オープンで初心者向けの issue。 Hacktoberfest の issue を見る

[ET-VK][Adreno] Low-rate non-deterministic CNN output, dependent on GPU power state; clean on Mali

オープン
  • 何か問題がありますか？
#21,939 コメント 1 件 リアクション 0 件 担当者 1 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

@SS-JIA がすでに取り組んでいます。

2026年8月24日 から。

評価

この issue はまだ評価されていません。

説明

module: vulkan
🐛 Describe the bug

On Adreno, CNNs lowered to the Vulkan delegate produce different results run to run, at a low but persistent rate, even with the im2col/GEMM path from #21938 disabled. EfficientNetV2-S gives 24 distinct results in 25 replays of the same already-encoded command buffer inside a single process, which rules out load-time state, prepack and weight packing.

Deviation magnitude: EfficientNetV2-S logits span [-1.44, 2.75]; typical deviations are 0.001 to 0.009, with rare excursions to about 4.0. argmax was stable across 30 runs for one input, so class flips need the rare large excursions.

This is very likely an Adreno driver or hardware problem, not an ExecuTorch bug
  1. A different GPU vendor is clean. Same binaries, same .pte, Galaxy S10+ (Exynos 9820, Mali-G76): 20 runs, 4020 in-process replays, zero deviations. The probe below that deviates on Adreno gives 0 across 1200 replays there. Output is also numerically correct on Mali, not merely stable: matched Vulkan and CPU exports from one model instance both land within fp32 rounding of the eager reference (9.9e-07 and 3.6e-07).
    Confirmed at the application level as well, not just with executor_runner: a demo app built from the stock Vulkan backend runs both object detection (YOLO26) and image classification (EfficientNetV2-S) on the S10 with correct and stable results.

Separately, and worth knowing for anyone weighing the Vulkan delegate on Mali: on this S10 the Vulkan delegate is close to 2x slower than the XNNPACK equivalent for the same models. That is the opposite of the Adreno device, where Vulkan is clearly faster. The S10 is a 2019 part so this is not necessarily representative of current Mali hardware, but it does mean the delegate is fast on the GPU where it is incorrect and slow on the GPU where it is correct.

  1. Khronos synchronization validation reports zero hazards on the failing workload. Layer injected into a debuggable app via gpu_debug_layers, sync validation confirmed active by the layer's own settings message. No SYNC-HAZARD, no VUID errors of any kind.
  2. Unconditional full memory barriers change nothing. Patching CommandBuffer::insert_barrier to always emit a global ALL_COMMANDS -> ALL_COMMANDS barrier with VK_ACCESS_MEMORY_READ_BIT | VK_ACCESS_MEMORY_WRITE_BIT before every dispatch, on top of the computed per-resource barriers, does not help.

The dispatch sequence is also deterministic: an env-gated trace in DispatchNode::encode gives byte-identical traces across runs.

Measurement note, important for anyone reproducing

The failure rate is strongly dependent on GPU power state and swings by more than an order of magnitude for the same .pte and same binary within minutes. Same build, EfficientNetV2-S: 79 distinct / 101 at ~1000 MHz, 13 / 101 at ~826 MHz, 1 / 101 pinned at 461 MHz.

adb shell cmd power set-fixed-performance-mode-enabled true pins the GPU at 461 MHz without root on this device (devfreq/{min,max}_freq are unavailable). Over 4 alternating pairs of 100 in-process replays: 3 excess distinct / 404 pinned versus 30 / 404 unpinned.

Consequences for methodology:

  • Any A/B comparison that is not interleaved within a single round is unreliable.
  • Patch executor_runner to dump outputs on every iteration of --num_executions; stock writes only the last. This is far more sensitive than relaunching the process.
  • Probe models must be activation calibrated. Untrained stacked convs collapse the signal to ~1e-19, where "distinct output" counts last-bit noise on values carrying no information.
Smallest structure that shows it

12 rounds, order rotated each round, 200 in-process replays per run, calibrated probes, input (1, 48, 64, 64), im2col disabled. Excess distinct outputs per 2400 replays:

graph run A run B
conv3x3 -> SiLU -> conv1x1 43 18
conv3x3 -> ReLU -> conv1x1 3 11
conv3x3 -> Sigmoid -> conv1x1 - 2
conv3x3 -> conv1x1 1 -
conv3x3 -> SiLU -> conv3x3 0 0

conv -> SiLU -> conv3x3 is 0 across 4800 replays while the same graph with a 1x1 consumer is the worst case. With no activation at all, kernel size alone does nothing (0 to 6 per 2400 for all four combinations of 1x1/3x3 producer and consumer). Absolute rates are low, so treat this as a lead rather than a conclusion. It is only visible with DVFS active; pinned at 461 MHz all four collapse to the noise floor.

Forcing 1x1 convs away from Conv2dMethod::Pointwise into the generic sliding window shader does not fix it, so it is not shader identity. The same generic shader is clean at 3x3 and dirty at 1x1.

Ruled out, each tested

Memory aliasing (add_tensor(..., -1) instead of mem_obj_id(); made it worse), storage type, command buffer splitting (execute_threshold_node_count set to SIZE_MAX), descriptor pool recycling (descriptor_pool_safety_factor 1.25 to 64.0), caller threading, host cache coherency, node count.

Possibly the same underlying defect as #21938, which is also Adreno-only but is 100% reproducible and independent of clock.

Versions

ExecuTorch 1.4.1. Failing: Galaxy S26 Ultra (SM-S948B), Adreno 840v2. Clean: Galaxy S10+ (SM-G975F), Mali-G76.

主要言語
Python
スター
5.1k
フォーク
1.2k
平均マージ
1日 23時間
マージ済み PR（30日）
713

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

pytorch/executorch のほかの issue

pytorch/executorch の issue をすべて見る

似ている issue

Python の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。