Parametrized dtype overridden in test_autotuner; wrong variable in test_integration error message
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 78/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Quiet
- Tech stack
- python, pytorch
- Domain
- testing-qa
Research direction
Start with test/kernel/test_autotuner.py and inspect the three TestQuantFlow methods and their dtype assertions. Run pytest test/kernel/test_autotuner.py -v to confirm both dtype variants exercise their parametrized values. Then inspect test_integration.py around lines 641-642 and ensure the failure message reports the SQNR for the tested output; done means the parametrized tests cover both dtypes and the message uses the checked value.
Written by the indexing model from the issue text.
Description
Bug 1: Parametrized dtype silently overridden in test_autotuner.py
All three test methods in TestQuantFlow are parametrized over dtype (both torch.bfloat16 and torch.float16), but each one immediately overwrites it with a hardcoded dtype = torch.bfloat16:
| Method | Line | Override |
|---|---|---|
test_int_mm |
40 | dtype = torch.bfloat16 |
test_int_mm_float8 |
62 | dtype = torch.bfloat16 |
test_int_scaled_mm |
88 | dtype = torch.bfloat16 |
This means the float16 test cases never actually test float16. They silently run with bfloat16 instead, giving false test coverage.
Additionally, test_int_scaled_mm asserts out32_1.dtype == torch.bfloat16 on line 96, which is hardcoded instead of using the parametrized dtype. If the override were removed without fixing this assertion, float16 tests would fail.
Bug 2: Wrong variable in error message in test_integration.py
Line 641-642:
assert SQNR(ref_f, test) > min_sqnr, (
f"got sqnr: {SQNR(ref_f, ref_q)}, expected: {min_sqnr}"
)
The assertion checks SQNR(ref_f, test), but the error message prints SQNR(ref_f, ref_q). ref_q is the compiled quantized reference computed earlier, not the value being tested. When this assertion fails, the error message shows the wrong SQNR value — the one between the float reference and the compiled quantized reference, not between the float reference and the actual test output.
Should be:
f"got sqnr: {SQNR(ref_f, test)}, expected: {min_sqnr}"
Reproduction
Bug 1: Run pytest test/kernel/test_autotuner.py -v and observe that test_int_mm_cuda_float16 and similar cases produce identical results to the bfloat16 variants.
Bug 2: No runtime failure — the bug only manifests in error messages when the assertion fails.
- Dominant language
- Python
- Stars
- 3k
- Forks
- 627
- Avg merge
- 4d 14h
- Merged PRs (30d)
- 34
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from pytorch/ao
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
Difficulty 1/5 Under an hour Newbie friendliness 92/100
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100