[BUG] DPA4C compression fails on RTX 4090 with CUDA: "pair_film matches 0 constants"

Abierto
  • ¿Algo no está bien?
#6,062 2 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 4 días

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
12/100
Tipo de issue
Error
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
python, pytorch
Área
machine-learning

Línea de trabajo

The failing check is _match_charge_state_constants() in deepmd/pt_expt/utils/serialization.py, reached during dp --pt-expt compress on a CUDA-exported graph-lower .pt2. Start by reproducing the freeze and compress steps with the attached checkpoint, then compare the constants in the CPU and CUDA exports of pair_film and type_embedding. Done means the compress step matches exactly one constant for pair_film and completes end to end, which the reporter has not confirmed on CPU yet.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

bug
Bug summary

I cannot compress a fine-tuned DPA4C model on an NVIDIA RTX 4090 using the CUDA backend.

Training, freezing, and validation inference succeed in the same environment. However, running:

dp --pt-expt compress -i frozen_RANDOM.pt2 -o compressed_RANDOM.pt2

fails during compressed-model export in _match_charge_state_constants() with:

RuntimeError: The compressed artifact 'pair_film' matches 0 constants of the exported lower; a runtime charge state needs exactly one so that the deployment knows which constant to overwrite.

Environment:

  • GPU: NVIDIA RTX 4090 (sm_89)
  • DeePMD-kit: v3.2.1.dev21+g1313650a7
  • Backend: pt_expt
  • PyTorch: 2.11.0+cu130
  • CUDA reported by PyTorch: 13.0
  • DP_CUDA_INFER=1

Previous inspection of the same model found exact matching pair_film and type_embedding constants in the CPU-exported graph, but not in the CUDA-exported graph. This comparison concerns constant matching; successful end-to-end CPU compression has not been confirmed.

Could you help identify why CUDA compression fails for this DPA4C model and recommend a fix or supported software combination?

This failure was observed in the environment above. Latest master has not yet been tested with this checkpoint.

DeePMD-kit Version

v3.2.1.dev21+g1313650a7 (commit 1313650a7ecd3ee29ccf5b3d04ff499e59248889)

Backend and its version

pt_expt; PyTorch 2.11.0+cu130; CUDA 13.0; NVIDIA RTX 4090 (sm_89)

How did you download the software?

Built from source

Input Files, Running Commands, Error Log, etc.
Model and environment
  • Model: fine-tuned DPA4C, initialized from DPA4C-Neo-OpenLAM-v20260902.pt.
  • GPU: NVIDIA RTX 4090 (sm_89).
  • DeePMD-kit: v3.2.1.dev21+g1313650a7.
  • Commit: 1313650a7ecd3ee29ccf5b3d04ff499e59248889.
  • Backend: pt_expt.
  • PyTorch: 2.11.0+cu130.
  • torch.version.cuda: 13.0.

Training, freezing, and validation inference succeeded in this environment. CUDA compression failed.

Commands
export DP_CUDA_INFER=1

dp --pt-expt freeze \
  -c model.ckpt.pt \
  -o frozen_RANDOM \
  --lower-kind graph

dp --pt-expt compress \
  -i frozen_RANDOM.pt2 \
  -o compressed_RANDOM.pt2
Compression error

The failure occurs in:
deepmd/pt_expt/utils/serialization.py
_match_charge_state_constants(...)

RuntimeError: The compressed artifact 'pair_film' matches 0 constants of the exported lower; a runtime charge state needs exactly one so that the deployment knows which constant to overwrite.
Previous CPU/CUDA export inspection

For the same model and DeePMD version:

  • pair_film: shape (14161, 64, 2), dtype float32.

    • CPU export: exact match with _tensor_constant4.
    • CUDA export: no same-shape tensor found in state_dict or constants.

  • type_embedding: shape (119, 64), dtype float32.

    • CPU export: exact match with _tensor_constant6.
    • CUDA export: _param_constant0 has the same shape but different values; maximum absolute difference is 1.6850241422653198.

These observations concern exported constant matching. Successful end-to-end CPU compression has not been confirmed.

The full traceback, model configuration, and a failing model file/download link will be provided as reproduction materials.

D1_6022_model_config.zip
dpa4c_issue_20261009.zip

Steps to Reproduce

  1. Use a Linux machine with an allocated NVIDIA RTX 4090 GPU and the software versions listed above.

  2. Use the failing fine-tuned DPA4C checkpoint, model.ckpt.pt. Confirm that PyTorch detects the GPU:

    python -c "import torch; print(torch.__version__, torch.version.cuda, torch.cuda.is_available()); print(torch.cuda.get_device_name(0))"

  3. Enable CUDA inference and freeze the checkpoint as a graph-lower model:

    export DP_CUDA_INFER=1
dp --pt-expt freeze \
  -c model.ckpt.pt \
  -o frozen_RANDOM \
  --lower-kind graph

    Freezing succeeds and produces frozen_RANDOM.pt2.

  4. Compress the frozen model on the same GPU in the same environment:

    dp --pt-expt compress \
  -i frozen_RANDOM.pt2 \
  -o compressed_RANDOM.pt2

  5. Compression fails in _match_charge_state_constants() with:
    "The compressed artifact 'pair_film' matches 0 constants of the exported lower".

Expected behavior:
Compression completes and produces a valid compressed CUDA .pt2 model.

Actual behavior:
Compression/export aborts because the required charge-state artifact cannot be matched to an exported constant.

This failure was observed at commit 1313650a7ecd3ee29ccf5b3d04ff499e59248889. Latest master has not yet been tested with this checkpoint.

Further Information, Files, and Links

No response

Lenguaje dominante
Python
Estrellas
2.1k
Forks
662
Merge medio
5 d 19 h
PR fusionados (30 d)
22

Preparar el entorno

Abrir en Codespaces

Inicia el contenedor de desarrollo del proyecto en tu navegador, con tu propia cuenta de GitHub.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de deepmodeling/deepmd-kit

Todos los issues de deepmodeling/deepmd-kit

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.