Discrepancy in actual and expected output for subgraph test : batchNormalization options.axis=0 + gelu
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 15/100
- Issue type
- Bug
- Clarity
- Needs clarification
- Activity status
- Active
- Tech stack
- tensorflow
- Domain
- machine-learning, testing
Research direction
Start from the subgraph conformance test that chains batchNormalization with options.axis=0 into gelu, using the inputs and expected values in the issue. Reproduce the ULP mismatch at index 4 on the CPU backend, then compare against the other backends the reporter tried. The thread asks the working group whether the tolerance or the expected values should change, so there is no code change to make until that decision is recorded. Done means a documented decision on the tolerance, plus either an updated threshold or a fix.
Written by the indexing model from the issue text.
Description
Issue
I was running this conformance test for NVIDIA's TensorRT-RTX EP with ORT backend and encountered a failure.
Input:
bnInput shape [3,1,2] float32 = [-1, 0, 1, 2, 3, 4]
bnMean shape [3] float32 = [0, 3, 6]
bnVariance shape [3] float32 = [1.0, 1.5, 2.0]
Graph:
batchNormalization(input=bnInput, mean=bnMean, variance=bnVariance, {axis: 0}) -> bnOutput
gelu(bnOutput) -> output
Expected output: shape [3,1,2] float32
[-0.15865567326545715, 0, -0.08366739749908447,
-0.16910307109355927, -0.03595117852091789, -0.1112278625369072]
Error happens at index 4
actual -0.035951320081949234 should be close enough to expected -0.03595117852091789 by ULP distance: expected <= 24 but got 38
Observations
I tested this with other execution providers in ORT and other backends and encountered similar or same error.
| Backend | Actual | ULP Difference |
|---|---|---|
| ORT CPU EP | −0.035951320081949234 | 38 |
| TFLite | −0.035951387137174606 | 56 |
| LiteRT (GPU) | −0.035951387137174606 | 56 |
Question for the Working Group
- What is the expected behavior for this test ?
- Should the discrepancy between the actual and expected outputs be investigated, or is the current error tolerance too strict and in need of adjustment?
- Dominant language
- Bikeshed
- Stars
- 566
- Forks
- 72
- Avg merge
- 6d 5h
- Merged PRs (30d)
- 1
Getting set up
- No Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from webmachinelearning/webnn
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
webmachinelearning/webnn#967 ·
-
Difficulty 5/5 Over a week Newbie friendliness 25/100
webmachinelearning/webnn#966 · 1 comment ·
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
webmachinelearning/webnn#965 · 1 comment ·
-
Difficulty 3/5 1-2 days Newbie friendliness 68/100
webmachinelearning/webnn#964 · 1 comment · 1 reaction ·
-
Difficulty 5/5 Over a week Newbie friendliness 30/100
webmachinelearning/webnn#963 · 2 comments · 1 reaction ·
All issues in webmachinelearning/webnn
Similar issues
-
Random Dropout MSEOpen
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
JacksonBurns/how-to-train-your-chemeleon#9 · 1 comment ·
-
new_case_failure skipped skipped_bmg
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
intel/torch-xpu-ops#5800 ·
Maintainers usually reply within 2 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
artlite-project/artlite#5 · 1 comment ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Maintainers usually reply within 1 day
-
CUDA OOM in omnivoice-demo voice cloning: create_voice_clone_prompt() runs without inference_modePossibly taken @karthik24iyer claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100