WebGPU ConvTranspose produces incorrect results for fp16 models when output spatial dim > 2048 (index math done in f16)

Open Beginner friendly
#28,976 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
78/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
cpp

Research direction

Start in onnxruntime/core/providers/webgpu/nn/conv_backprop.cc at ConvTranspose2DProgram::GenerateShaderCode, focusing on the dyR and dyC index calculations and bounds checks. Reproduce the fp16 ConvTranspose case with an output spatial dimension above 2048 and compare WebGPU against the CPU EP. Done means large fp16 outputs agree with CPU while existing fp32 behavior remains correct.

Written by the indexing model from the issue text.

Description

ep:WebGPU platform:web stale

Describe the issue

The WebGPU EP's ConvTranspose kernel produces incorrect output for float16 models once an output spatial dimension exceeds ~2048. The error is data-independent and size-dependent: it grows with output size and the first incorrect output element appears around spatial position 2048. The same model run on the CPU EP (or on WebGPU in fp32) is correct.

This is not the same as the already-merged fix #27749 (non-vectorizable input channels); that code path is present in 1.25/1.26/main and does not cover this case.

Root cause

In onnxruntime/core/providers/webgpu/nn/conv_backprop.cc, ConvTranspose2DProgram::GenerateShaderCode computes the input-gradient indices dyR/dyC and their bounds checks using dy_element_t, which is the tensor's element type — f16 for an fp16 model:

// lines ~108-110
"let dyR = (dy_element_t(dyRCorner) + dy_element_t(wR)) / dy_element_t(uniforms.strides[0]);\n"
...
"if (dyR < 0.0 || dyR >= dy_element_t(uniforms.dy_shape[" << row_dim << "]) || fract(dyR) > 0.0 || wRPerm < 0) {\n"

// lines ~123-126 (same pattern for the column axis)
"let dyC = (dy_element_t(dyCCorner) + dy_element_t(wC)) / dy_element_t(uniforms.strides.y);\n"
...
"if (dyC < 0.0 || dyC >= dy_element_t(uniforms.dy_shape[" << col_dim << "]) ||\n"
"    fract(dyC) > 0.0 || wCPerm < 0) {\n"

f16 cannot represent consecutive integers exactly above 2048 (11-bit mantissa). For output positions beyond that range, dy_element_t(dyRCorner), the division, the fract() divisibility test, and the >= dy_shape bounds check all operate on rounded values. This selects the wrong input element (or wrongly passes/fails the stride-divisibility and bounds checks), corrupting a band of outputs. Because it is integer-index arithmetic forced through f16, the defect depends only on size, not on the data.

Reproduction

  • Any fp16 model containing a ConvTranspose whose output spatial dimension is > ~2048 (a 1-D ConvTranspose expressed as 2-D with H=1 reproduces it; e.g. in_channels=128, kernel=11, stride=5, input length ~1600 → output length ~8000).
  • Run the node on the WebGPU EP vs the CPU EP on identical inputs and compare.

Observed relative error vs CPU as a function of output spatial size (single ConvTranspose, fp16):

output spatial dim rel. error
≤ 996 ~0% (correct)
3996 ~31%
7996 ~47%
15996 ~78%

First incorrect element appears at output position ≈ 2048, consistent with the f16 integer-precision limit.

Confirmed on: ORT 1.25.0 and 1.26.0 (byte-identical divergence between the two), and the bug is still present on current main (a1fc916579). git log v1.26.0..main contains no WebGPU conv_transpose fix.

Environment

  • EP: WebGPU (Dawn, Vulkan backend)
  • GPU tested: Intel Iris Xe (Vulkan), Linux x64
  • Affected dtype: float16 (fp32 models are unaffected)

Suggested fix

Compute dyR/dyC and their bounds checks in f32 rather than dy_element_t. The index arithmetic is logically integer and should not be done in the storage element type. Four edits in GenerateShaderCode (the dyR =, dyR >=, dyC =, dyC >= expressions):

let dyR = (f32(dyRCorner) + f32(wR)) / f32(uniforms.strides[0]);
...
if (dyR < 0.0 || dyR >= f32(uniforms.dy_shape[row_dim]) || fract(dyR) > 0.0 || wRPerm < 0) {
...
let dyC = (f32(dyCCorner) + f32(wC)) / f32(uniforms.strides.y);
...
if (dyC < 0.0 || dyC >= f32(uniforms.dy_shape[col_dim]) || fract(dyC) > 0.0 || wCPerm < 0) {

After this change (ORT 1.26, Intel Vulkan WebGPU): the size sweep above drops to a uniform ~0.31% across all sizes, and an isolated real-data ConvTranspose that was ~23% off drops to ~0.28%. The residual is ordinary fp16 drift, not a kernel defect.

Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
3d 23h
Merged PRs (30d)
172

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from microsoft/onnxruntime

All issues in microsoft/onnxruntime

Similar issues

More C++ issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.