WebGPU ConvTranspose produces incorrect results for fp16 models when output spatial dim > 2048 (index math done in f16)
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 78/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Active
- Tech stack
- cpp
- Domain
- backend, machine-learning
Research direction
Start in onnxruntime/core/providers/webgpu/nn/conv_backprop.cc at ConvTranspose2DProgram::GenerateShaderCode, focusing on the dyR and dyC index calculations and bounds checks. Reproduce the fp16 ConvTranspose case with an output spatial dimension above 2048 and compare WebGPU against the CPU EP. Done means large fp16 outputs agree with CPU while existing fp32 behavior remains correct.
Written by the indexing model from the issue text.
Description
Describe the issue
The WebGPU EP's ConvTranspose kernel produces incorrect output for float16 models once an output spatial dimension exceeds ~2048. The error is data-independent and size-dependent: it grows with output size and the first incorrect output element appears around spatial position 2048. The same model run on the CPU EP (or on WebGPU in fp32) is correct.
This is not the same as the already-merged fix #27749 (non-vectorizable input channels); that code path is present in 1.25/1.26/main and does not cover this case.
Root cause
In onnxruntime/core/providers/webgpu/nn/conv_backprop.cc, ConvTranspose2DProgram::GenerateShaderCode computes the input-gradient indices dyR/dyC and their bounds checks using dy_element_t, which is the tensor's element type — f16 for an fp16 model:
// lines ~108-110
"let dyR = (dy_element_t(dyRCorner) + dy_element_t(wR)) / dy_element_t(uniforms.strides[0]);\n"
...
"if (dyR < 0.0 || dyR >= dy_element_t(uniforms.dy_shape[" << row_dim << "]) || fract(dyR) > 0.0 || wRPerm < 0) {\n"
// lines ~123-126 (same pattern for the column axis)
"let dyC = (dy_element_t(dyCCorner) + dy_element_t(wC)) / dy_element_t(uniforms.strides.y);\n"
...
"if (dyC < 0.0 || dyC >= dy_element_t(uniforms.dy_shape[" << col_dim << "]) ||\n"
" fract(dyC) > 0.0 || wCPerm < 0) {\n"
f16 cannot represent consecutive integers exactly above 2048 (11-bit mantissa). For output positions beyond that range, dy_element_t(dyRCorner), the division, the fract() divisibility test, and the >= dy_shape bounds check all operate on rounded values. This selects the wrong input element (or wrongly passes/fails the stride-divisibility and bounds checks), corrupting a band of outputs. Because it is integer-index arithmetic forced through f16, the defect depends only on size, not on the data.
Reproduction
- Any fp16 model containing a
ConvTransposewhose output spatial dimension is > ~2048 (a 1-D ConvTranspose expressed as 2-D with H=1 reproduces it; e.g. in_channels=128, kernel=11, stride=5, input length ~1600 → output length ~8000). - Run the node on the WebGPU EP vs the CPU EP on identical inputs and compare.
Observed relative error vs CPU as a function of output spatial size (single ConvTranspose, fp16):
| output spatial dim | rel. error |
|---|---|
| ≤ 996 | ~0% (correct) |
| 3996 | ~31% |
| 7996 | ~47% |
| 15996 | ~78% |
First incorrect element appears at output position ≈ 2048, consistent with the f16 integer-precision limit.
Confirmed on: ORT 1.25.0 and 1.26.0 (byte-identical divergence between the two), and the bug is still present on current main (a1fc916579). git log v1.26.0..main contains no WebGPU conv_transpose fix.
Environment
- EP: WebGPU (Dawn, Vulkan backend)
- GPU tested: Intel Iris Xe (Vulkan), Linux x64
- Affected dtype: float16 (fp32 models are unaffected)
Suggested fix
Compute dyR/dyC and their bounds checks in f32 rather than dy_element_t. The index arithmetic is logically integer and should not be done in the storage element type. Four edits in GenerateShaderCode (the dyR =, dyR >=, dyC =, dyC >= expressions):
let dyR = (f32(dyRCorner) + f32(wR)) / f32(uniforms.strides[0]);
...
if (dyR < 0.0 || dyR >= f32(uniforms.dy_shape[row_dim]) || fract(dyR) > 0.0 || wRPerm < 0) {
...
let dyC = (f32(dyCCorner) + f32(wC)) / f32(uniforms.strides.y);
...
if (dyC < 0.0 || dyC >= f32(uniforms.dy_shape[col_dim]) || fract(dyC) > 0.0 || wCPerm < 0) {
After this change (ORT 1.26, Intel Vulkan WebGPU): the size sweep above drops to a uniform ~0.31% across all sizes, and an isolated real-data ConvTranspose that was ~23% off drops to ~0.28%. The residual is ordinary fp16 drift, not a kernel defect.
- Dominant language
- C++
- Stars
- 21.9k
- Forks
- 4.2k
- Avg merge
- 3d 23h
- Merged PRs (30d)
- 172
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from microsoft/onnxruntime
-
ep:MIGraphX
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
microsoft/onnxruntime#32352 · 1 comment ·
-
`optimization.disable_specified_optimizers` is documented as comma-separated but is parsed on `;` Open
Difficulty 1/5 Under an hour Newbie friendliness 88/100
microsoft/onnxruntime#32211 ·
-
stale
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
microsoft/onnxruntime#29750 · 3 comments ·
-
[Documentation] Python tutorial missing PyTorch export guidance and external data file handling Opendocumentation
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
microsoft/onnxruntime#28763 · 5 comments ·
-
[Documentation] Add C++ in-code profiling example code to the "Profiling Tools" documentation page Opendocumentation
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
microsoft/onnxruntime#28512 · 4 comments · 1 reaction ·
All issues in microsoft/onnxruntime
Similar issues
-
Difficulty 1/5 Under an hour Newbie friendliness 90/100
AXERA-TECH/ax-llm#77 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
-
bug-unconfirmed
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
NVIDIA/cuda-samples#453 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
infiniflow/infinity#3502 ·