Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Finite FP16 MX blocks become NaNs on the default PyTorch path

Open
  • Is something wrong?
#46 0 comments 0 reactions 0 assignees View on GitHub

@vera-rubin is already working on this.

Since Oct 13, 2026.

  • #47 by @vera-rubin — open

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
48/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
python, pytorch
Domain
machine-learning

Research direction

Start with mx/mx_ops.py at the reported scale calculation around line 285, then inspect the zero guard around line 78. Run the provided reproduction on CPU and compare the six formats and the linear call with the expected finite outputs. Done means zero blocks stay zero and the small E5M2 block and public linear call return finite expected results; the report says no correction was tested.

Written by the indexing model from the issue text.

Description

Finite FP16 blocks become NaNs during MX quantization with custom_cuda=False. This includes zero blocks and a public linear call that should return zero.

Tested at microsoft/microxcaling commit 7bc41952de394f5cc5e782baf132e7c7542eb4e4 (mx==1.1.0). Both fresh environments used Python 3.12.3 and NumPy 1.26.4 on x86_64 Linux. The failure reproduced with CPU PyTorch 2.2.0+cpu and 2.4.0+cpu.

Reproduction

Run these commands in Bash with Python 3.12.3. The example is CC0.

git clone https://github.com/microsoft/microxcaling.git microxcaling
cd microxcaling
git checkout --detach 7bc41952de394f5cc5e782baf132e7c7542eb4e4
python3.12 -m venv .venv
.venv/bin/python -m pip install torch==2.4.0 --index-url https://download.pytorch.org/whl/cpu
.venv/bin/python -m pip install numpy==1.26.4 packaging ninja==1.13.2 hatchling==1.32.4 hatch-vcs==0.5.0 editables
MAX_JOBS=1 .venv/bin/python -m pip install --no-build-isolation --no-deps -e .
OMP_NUM_THREADS=1 MKL_NUM_THREADS=1 .venv/bin/python - <<'PY'
import torch
from mx import linear
from mx.mx_ops import quantize_mx_op
from mx.specs import MxSpecs

torch.set_num_threads(1)
specs = MxSpecs()
specs.update(scale_bits=8, block_size=32, custom_cuda=False)

formats = (
    "fp8_e5m2", "fp8_e4m3", "fp6_e3m2",
    "fp6_e2m3", "fp4_e2m1", "int8",
)
for fmt in formats:
    x = torch.zeros(32, dtype=torch.float16)
    y = quantize_mx_op(x, specs, fmt, axes=-1, round="even")
    print(fmt, "zero NaNs:", y.isnan().sum().item())

x = torch.zeros(32, dtype=torch.float16)
x[0] = 2**-10
y = quantize_mx_op(x, specs, "fp8_e5m2", axes=-1, round="even")
print("E5M2 small:", y[:2].tolist())

specs.update(
    a_elem_format="fp8_e5m2", w_elem_format="fp8_e5m2",
    quantize_backprop=False,
)
output = linear(
    torch.zeros((1, 32), dtype=torch.float16),
    torch.ones((1, 32), dtype=torch.float16),
    mx_specs=specs,
)
print("linear:", output.item())
PY

For the other tested Torch version, use a fresh venv and substitute torch==2.2.0 in the install command. The no-dependency editable install omits unused torchvision and torchaudio, whose dependency mismatch is already reported in issue 31.

Expected and observed

The README documents native FP16 computation and CPU as the numeric reference. PR 28 added native FP16 support.

All-zero blocks should remain zero. For the small E5M2 block, the implemented scale algorithm selects X=2^-25 (E8M0 byte 0x66). The nonzero element is 2^15 (E5M2 code 0x78), so the decoded result is exactly 2^-10. Zero elements remain zero. OCP MX v1.0 §5.1 and Tables 2 and 7 define these finite encodings and products. The scale need not fit the FP16 container. An all-zero block does not require a unique scale.

These encodings are derived expectations for this library's maximum-based scale algorithm. The API returns dequantized tensors, and MX §6.3 permits other scale-selection algorithms.

Case Expected Observed on both Torch versions
Zero block, each of the six formats 32 zeros, no NaNs 32 NaNs
E5M2 block with 2^-10 and 31 zeros Identical finite block 32 NaNs, beginning [nan, nan]
FP16 zero input and unit weights in linear 0.0 nan

Source and scope

mx_ops.py:285 computes 2**shared_exp in FP16. The exact 2^-25 scale rounds to zero before division. Zero blocks also lose the FP32 zero guard at mx_ops.py:78 and underflow the clamped scale. Scale and normalization intermediates need a type that represents these values. No correction was tested.

Zero blocks were checked in all six formats with even, nearest, floor and both FP32-flush settings. The small input is an exact power of two. Wider FP32/BF16 inputs and the FP16 2^-9 threshold control passed. The original C++ CPU numeric code returned the expected values through a CPU-only build adapter. The adapter preserved the numeric code and changed only build plumbing. The stock CUDA extension and GPU execution were not tested. These cases use finite inputs and forward computations.

Found by ChipLab's AI verification agents and independently reproduced.

ChipLab ref: CLAB-2026-623

Dominant language
Python
Stars
365
Forks
54
No merged PRs in 30d

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from microsoft/microxcaling

All issues in microsoft/microxcaling

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.