Parametrized dtype overridden in test_autotuner; wrong variable in test_integration error message

Open Beginner friendly
#4,339 0 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
78/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Quiet
Tech stack
python, pytorch
Domain
testing-qa

Research direction

Start with test/kernel/test_autotuner.py and inspect the three TestQuantFlow methods and their dtype assertions. Run pytest test/kernel/test_autotuner.py -v to confirm both dtype variants exercise their parametrized values. Then inspect test_integration.py around lines 641-642 and ensure the failure message reports the SQNR for the tested output; done means the parametrized tests cover both dtypes and the message uses the checked value.

Written by the indexing model from the issue text.

Description

Bug 1: Parametrized dtype silently overridden in test_autotuner.py

All three test methods in TestQuantFlow are parametrized over dtype (both torch.bfloat16 and torch.float16), but each one immediately overwrites it with a hardcoded dtype = torch.bfloat16:

Method Line Override
test_int_mm 40 dtype = torch.bfloat16
test_int_mm_float8 62 dtype = torch.bfloat16
test_int_scaled_mm 88 dtype = torch.bfloat16

This means the float16 test cases never actually test float16. They silently run with bfloat16 instead, giving false test coverage.

Additionally, test_int_scaled_mm asserts out32_1.dtype == torch.bfloat16 on line 96, which is hardcoded instead of using the parametrized dtype. If the override were removed without fixing this assertion, float16 tests would fail.

Bug 2: Wrong variable in error message in test_integration.py

Line 641-642:

assert SQNR(ref_f, test) > min_sqnr, (
    f"got sqnr: {SQNR(ref_f, ref_q)}, expected: {min_sqnr}"
)

The assertion checks SQNR(ref_f, test), but the error message prints SQNR(ref_f, ref_q). ref_q is the compiled quantized reference computed earlier, not the value being tested. When this assertion fails, the error message shows the wrong SQNR value — the one between the float reference and the compiled quantized reference, not between the float reference and the actual test output.

Should be:

f"got sqnr: {SQNR(ref_f, test)}, expected: {min_sqnr}"

Reproduction

Bug 1: Run pytest test/kernel/test_autotuner.py -v and observe that test_int_mm_cuda_float16 and similar cases produce identical results to the bfloat16 variants.

Bug 2: No runtime failure — the bug only manifests in error messages when the assertion fails.

Dominant language
Python
Stars
3k
Forks
627
Avg merge
4d 14h
Merged PRs (30d)
34

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from pytorch/ao

All issues in pytorch/ao

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.