[BUG] DPA4C compression fails on RTX 4090 with CUDA: "pair_film matches 0 constants"

Đang mở
  • Có gì không ổn?
#6,062 2 bình luận 0 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 3 ngày

Chưa có ai nhận issue này.

Đánh giá

Độ khó
5/5
Thời gian dự kiến
Hơn một tuần
Mức phù hợp với người mới
12/100
Loại issue
Lỗi
Độ rõ ràng
Khá rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
python, pytorch
Lĩnh vực
machine-learning

Hướng nghiên cứu

The failing check is _match_charge_state_constants() in deepmd/pt_expt/utils/serialization.py, reached during dp --pt-expt compress on a CUDA-exported graph-lower .pt2. Start by reproducing the freeze and compress steps with the attached checkpoint, then compare the constants in the CPU and CUDA exports of pair_film and type_embedding. Done means the compress step matches exactly one constant for pair_film and completes end to end, which the reporter has not confirmed on CPU yet.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

bug
Bug summary

I cannot compress a fine-tuned DPA4C model on an NVIDIA RTX 4090 using the CUDA backend.

Training, freezing, and validation inference succeed in the same environment. However, running:

dp --pt-expt compress -i frozen_RANDOM.pt2 -o compressed_RANDOM.pt2

fails during compressed-model export in _match_charge_state_constants() with:

RuntimeError: The compressed artifact 'pair_film' matches 0 constants of the exported lower; a runtime charge state needs exactly one so that the deployment knows which constant to overwrite.

Environment:

  • GPU: NVIDIA RTX 4090 (sm_89)
  • DeePMD-kit: v3.2.1.dev21+g1313650a7
  • Backend: pt_expt
  • PyTorch: 2.11.0+cu130
  • CUDA reported by PyTorch: 13.0
  • DP_CUDA_INFER=1

Previous inspection of the same model found exact matching pair_film and type_embedding constants in the CPU-exported graph, but not in the CUDA-exported graph. This comparison concerns constant matching; successful end-to-end CPU compression has not been confirmed.

Could you help identify why CUDA compression fails for this DPA4C model and recommend a fix or supported software combination?

This failure was observed in the environment above. Latest master has not yet been tested with this checkpoint.

DeePMD-kit Version

v3.2.1.dev21+g1313650a7 (commit 1313650a7ecd3ee29ccf5b3d04ff499e59248889)

Backend and its version

pt_expt; PyTorch 2.11.0+cu130; CUDA 13.0; NVIDIA RTX 4090 (sm_89)

How did you download the software?

Built from source

Input Files, Running Commands, Error Log, etc.
Model and environment
  • Model: fine-tuned DPA4C, initialized from DPA4C-Neo-OpenLAM-v20260902.pt.
  • GPU: NVIDIA RTX 4090 (sm_89).
  • DeePMD-kit: v3.2.1.dev21+g1313650a7.
  • Commit: 1313650a7ecd3ee29ccf5b3d04ff499e59248889.
  • Backend: pt_expt.
  • PyTorch: 2.11.0+cu130.
  • torch.version.cuda: 13.0.

Training, freezing, and validation inference succeeded in this environment. CUDA compression failed.

Commands
export DP_CUDA_INFER=1

dp --pt-expt freeze \
  -c model.ckpt.pt \
  -o frozen_RANDOM \
  --lower-kind graph

dp --pt-expt compress \
  -i frozen_RANDOM.pt2 \
  -o compressed_RANDOM.pt2
Compression error

The failure occurs in:
deepmd/pt_expt/utils/serialization.py
_match_charge_state_constants(...)

RuntimeError: The compressed artifact 'pair_film' matches 0 constants of the exported lower; a runtime charge state needs exactly one so that the deployment knows which constant to overwrite.
Previous CPU/CUDA export inspection

For the same model and DeePMD version:

  • pair_film: shape (14161, 64, 2), dtype float32.

    • CPU export: exact match with _tensor_constant4.
    • CUDA export: no same-shape tensor found in state_dict or constants.

  • type_embedding: shape (119, 64), dtype float32.

    • CPU export: exact match with _tensor_constant6.
    • CUDA export: _param_constant0 has the same shape but different values; maximum absolute difference is 1.6850241422653198.

These observations concern exported constant matching. Successful end-to-end CPU compression has not been confirmed.

The full traceback, model configuration, and a failing model file/download link will be provided as reproduction materials.

D1_6022_model_config.zip
dpa4c_issue_20261009.zip

Steps to Reproduce

  1. Use a Linux machine with an allocated NVIDIA RTX 4090 GPU and the software versions listed above.

  2. Use the failing fine-tuned DPA4C checkpoint, model.ckpt.pt. Confirm that PyTorch detects the GPU:

    python -c "import torch; print(torch.__version__, torch.version.cuda, torch.cuda.is_available()); print(torch.cuda.get_device_name(0))"

  3. Enable CUDA inference and freeze the checkpoint as a graph-lower model:

    export DP_CUDA_INFER=1
dp --pt-expt freeze \
  -c model.ckpt.pt \
  -o frozen_RANDOM \
  --lower-kind graph

    Freezing succeeds and produces frozen_RANDOM.pt2.

  4. Compress the frozen model on the same GPU in the same environment:

    dp --pt-expt compress \
  -i frozen_RANDOM.pt2 \
  -o compressed_RANDOM.pt2

  5. Compression fails in _match_charge_state_constants() with:
    "The compressed artifact 'pair_film' matches 0 constants of the exported lower".

Expected behavior:
Compression completes and produces a valid compressed CUDA .pt2 model.

Actual behavior:
Compression/export aborts because the required charge-state artifact cannot be matched to an exported constant.

This failure was observed at commit 1313650a7ecd3ee29ccf5b3d04ff499e59248889. Latest master has not yet been tested with this checkpoint.

Further Information, Files, and Links

No response

Ngôn ngữ chính
Python
Star
2.1k
Fork
664
Merge trung bình
4 ngày 23 giờ
Pull request đã merge (30 ngày)
25

Chuẩn bị môi trường

Mở trong Codespaces

Khởi chạy dev container của dự án ngay trên trình duyệt, bằng tài khoản GitHub của bạn.

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của deepmodeling/deepmd-kit

Tất cả issue của deepmodeling/deepmd-kit

Issue tương tự

Thêm issue về Python

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.