Hacktoberfest 2026: những issue maintainer đã đánh dấu cho tháng Mười, đang mở và phù hợp người mới. Xem issue Hacktoberfest

[Bug] init_quantized_weights / --low_memory_mode exports numerically broken NVFP4 (quantizes meta tensors before weights load)

Đang mở
  • Có gì không ổn?
#2,160 1 bình luận 0 reaction 1 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 2 ngày

@cjluo-nv đang làm issue này rồi.

Từ ngày 17/8/2026.

Đánh giá

Issue này chưa được đánh giá.

Mô tả

Summary

init_quantized_weights (public API, exported in modelopt/torch/quantization/plugins/accelerate.py:__all__, and the mechanism behind hf_ptq.py --low_memory_mode) produces an NVFP4 checkpoint that is silently wrong: half-sized weight_scale tensors, dequantization cosine similarity 0.756 against the source BF16 (healthy NVFP4 is >0.99), and a checkpoint vLLM refuses to load.

There is no error at export time. The run completes and prints Quantized model exported to: ....

Root cause

In patched_from_pretrained (accelerate.py, current main a21173a4):

with init_empty_weights():
    model = cls.from_config(config, dtype=torch_dtype)   # L230 — meta tensors

mtq.quantize(model, quant_cfg)                            # L232 — quantizes META
mtq.compress(model, config=mtq.CompressConfig(...))       # L233 — packs META
_device_map = get_model_device_map(model, gpu_mem_percentage)

return load_checkpoint_and_dispatch(                      # L236 — real weights load
    model, checkpoint=..., device_map=_device_map, *args, **kwargs,
)

Quantization and compression run against init_empty_weights() meta tensors, i.e. before any real weight values exist. Calibration/absmax has nothing to measure, and mtq.compress has already replaced each Linear.weight with a packed uint8 buffer — so load_checkpoint_and_dispatch then loads real BF16 values into buffers whose shape and scale metadata describe the packed view.

Observable consequence on a Qwen2 model: k_proj.weight_scale is exported as [128, 28] where NVFP4 block-16 over 896 in-features requires [128, 56] — the scale grid was computed over 448 packed bytes instead of 896 logical values. vLLM aborts at load:

RuntimeError: start (0) + length (304) exceeds dimension size (152).

Dequantizing the export by hand and comparing to the BF16 source gives cosine 0.756 (lo/hi nibble order; 0.015 with nibbles swapped, 0.758 assuming block-16-truncated) — the numbers are damaged at quantization time, not at load time, so no loader-side fix helps.

Reproduction (one command, 0.5B model — no large hardware needed)

# inside nvcr.io/nvidia/vllm:26.05.post1-py3, nvidia-modelopt[hf]==0.43.0
python hf_ptq.py \
  --pyt_ckpt_path ./Qwen2.5-0.5B-Instruct \   # local dir; repo ids are rejected on this path
  --qformat nvfp4 --kv_cache_qformat none \
  --calib_size 512 --calib_seq 512 --batch_size 1 \
  --dataset ./calib.jsonl --attn_implementation sdpa \
  --skip_generate --low_memory_mode \
  --export_path ./out-NVFP4

Then check the export: weight_scale second dimension is half of in_features/16, and ~168 runtime-state tensors named *.weight_quantizer._double_scale leak into the state dict (vLLM: no module or parameter named ... weight_quantizer).

Three smaller defects sit on the same path and block reaching the export at all; I have opened separate small PRs for the first two:

  1. patched_from_pretrained forwards **kwargs (including attn_implementation) into load_checkpoint_and_dispatch(), which does not accept it → TypeError.
  2. No model.tie_weights() before quantization, so tied lm_head.weight stays on meta and dispatch_model's .to() raises Cannot copy out of meta tensor (accelerate's docs require tie_weights() before load_checkpoint_and_dispatch).
  3. The path accepts only local checkpoint directories, not HF repo ids (ValueError from load_checkpoint_in_model) — worth documenting if intentional.

Environment

modelopt 0.43.0, NGC nvcr.io/nvidia/vllm:26.05.post1-py3, NVIDIA GB10 (SM121, aarch64), 121 GB unified memory. Verified the same code path is unchanged on current main (a21173a4).

Impact / workaround

This is the documented OOM fallback for models larger than device memory, so it is exactly the path users reach for when they have no alternative. For a 72B model (145 GB BF16 on a 121 GB box) I ended up going through llm-compressor with accelerate disk offload instead, which produced a correct checkpoint. Full write-up with the dequantization check and all artifacts: https://github.com/spped2000/thaillm-nvfp4-dgx-spark/blob/main/results/quant72b/FINDING_modelopt_lowmem_broken.md

Suggested direction: load real weights before mtq.quantize/mtq.compress (e.g. dispatch first with an offload folder, then quantize sequentially), or fail loudly when quantization is attempted on meta tensors.

Disclosure: investigated with assistance from Claude (Anthropic); every number above comes from real runs on the hardware listed.

Ngôn ngữ chính
Python
Star
5.4k
Fork
767
Merge trung bình
3 ngày 11 giờ
Pull request đã merge (30 ngày)
203

Chuẩn bị môi trường

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của NVIDIA/Model-Optimizer

Tất cả issue của NVIDIA/Model-Optimizer

Issue tương tự

Thêm issue về Python

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.