Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

[Bug] init_quantized_weights / --low_memory_mode exports numerically broken NVFP4 (quantizes meta tensors before weights load)

Aperta
  • Qualcosa non va?
#2,160 1 commento 0 reazioni 1 assegnatario Vedi su GitHub

I maintainer di solito rispondono entro 2 giorni

@cjluo-nv ci sta già lavorando.

Dal 17/8/2026.

Valutazione

Questa issue non è ancora stata valutata.

Descrizione

Summary

init_quantized_weights (public API, exported in modelopt/torch/quantization/plugins/accelerate.py:__all__, and the mechanism behind hf_ptq.py --low_memory_mode) produces an NVFP4 checkpoint that is silently wrong: half-sized weight_scale tensors, dequantization cosine similarity 0.756 against the source BF16 (healthy NVFP4 is >0.99), and a checkpoint vLLM refuses to load.

There is no error at export time. The run completes and prints Quantized model exported to: ....

Root cause

In patched_from_pretrained (accelerate.py, current main a21173a4):

with init_empty_weights():
    model = cls.from_config(config, dtype=torch_dtype)   # L230 — meta tensors

mtq.quantize(model, quant_cfg)                            # L232 — quantizes META
mtq.compress(model, config=mtq.CompressConfig(...))       # L233 — packs META
_device_map = get_model_device_map(model, gpu_mem_percentage)

return load_checkpoint_and_dispatch(                      # L236 — real weights load
    model, checkpoint=..., device_map=_device_map, *args, **kwargs,
)

Quantization and compression run against init_empty_weights() meta tensors, i.e. before any real weight values exist. Calibration/absmax has nothing to measure, and mtq.compress has already replaced each Linear.weight with a packed uint8 buffer — so load_checkpoint_and_dispatch then loads real BF16 values into buffers whose shape and scale metadata describe the packed view.

Observable consequence on a Qwen2 model: k_proj.weight_scale is exported as [128, 28] where NVFP4 block-16 over 896 in-features requires [128, 56] — the scale grid was computed over 448 packed bytes instead of 896 logical values. vLLM aborts at load:

RuntimeError: start (0) + length (304) exceeds dimension size (152).

Dequantizing the export by hand and comparing to the BF16 source gives cosine 0.756 (lo/hi nibble order; 0.015 with nibbles swapped, 0.758 assuming block-16-truncated) — the numbers are damaged at quantization time, not at load time, so no loader-side fix helps.

Reproduction (one command, 0.5B model — no large hardware needed)

# inside nvcr.io/nvidia/vllm:26.05.post1-py3, nvidia-modelopt[hf]==0.43.0
python hf_ptq.py \
  --pyt_ckpt_path ./Qwen2.5-0.5B-Instruct \   # local dir; repo ids are rejected on this path
  --qformat nvfp4 --kv_cache_qformat none \
  --calib_size 512 --calib_seq 512 --batch_size 1 \
  --dataset ./calib.jsonl --attn_implementation sdpa \
  --skip_generate --low_memory_mode \
  --export_path ./out-NVFP4

Then check the export: weight_scale second dimension is half of in_features/16, and ~168 runtime-state tensors named *.weight_quantizer._double_scale leak into the state dict (vLLM: no module or parameter named ... weight_quantizer).

Three smaller defects sit on the same path and block reaching the export at all; I have opened separate small PRs for the first two:

  1. patched_from_pretrained forwards **kwargs (including attn_implementation) into load_checkpoint_and_dispatch(), which does not accept it → TypeError.
  2. No model.tie_weights() before quantization, so tied lm_head.weight stays on meta and dispatch_model's .to() raises Cannot copy out of meta tensor (accelerate's docs require tie_weights() before load_checkpoint_and_dispatch).
  3. The path accepts only local checkpoint directories, not HF repo ids (ValueError from load_checkpoint_in_model) — worth documenting if intentional.

Environment

modelopt 0.43.0, NGC nvcr.io/nvidia/vllm:26.05.post1-py3, NVIDIA GB10 (SM121, aarch64), 121 GB unified memory. Verified the same code path is unchanged on current main (a21173a4).

Impact / workaround

This is the documented OOM fallback for models larger than device memory, so it is exactly the path users reach for when they have no alternative. For a 72B model (145 GB BF16 on a 121 GB box) I ended up going through llm-compressor with accelerate disk offload instead, which produced a correct checkpoint. Full write-up with the dequantization check and all artifacts: https://github.com/spped2000/thaillm-nvfp4-dgx-spark/blob/main/results/quant72b/FINDING_modelopt_lowmem_broken.md

Suggested direction: load real weights before mtq.quantize/mtq.compress (e.g. dispatch first with an offload folder, then quantize sequentially), or fail loudly when quantization is attempted on meta tensors.

Disclosure: investigated with assistance from Claude (Anthropic); every number above comes from real runs on the hardware listed.

Lingua principale
Python
Stelle
5.3k
Fork
745
Merge medio
4g 2h
PR unite (30g)
160

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di NVIDIA/Model-Optimizer

Tutte le issue di NVIDIA/Model-Optimizer

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.