[BUG] deepmd-kit v3.2.0 installed with conda cannot freeze fine-tuned model checkpoint

Start with deepmd/pt/entrypoints/main.py and deepmd/pt/entrypoints/freeze_pt2.py, especially the freeze and freeze_sezm_to_pt2 entry points shown in the traceback. Reproduce the documented conda installation and dp --pt freeze command, then investigate the missing cuda.h error. Done means a fine-tuned checkpoint freezes successfully and produces the requested output.

Bug summary

Recently I installed deepmd-kit v3.2.0 via conda, using command

conda create -n deepmd deepmd-kit lammps horovod -c conda-forge

as indicated here in the documentation

However, when attempting to freeze a fine-tuned DPA-4 model checkpoint with:

dp --pt freeze -c model.ckpt.pt -o frozen 

Deepmd exited with the following error:

nohup: ignoring input
To get the best performance, it is recommended to adjust the number of threads by setting the environment variables OMP_NUM_THREADS, DP_INTRA_OP_PARALLELISM_THREADS, and DP_INTER_OP_PARALLELISM_THREADS. See https://deepmd.rtfd.io/parallelism/ for more information.
/tmp/tmp8hu578aw/cuda_utils.c:1:10: fatal error: cuda.h: No such file or directory
    1 | #include "cuda.h"
      |          ^~~~~~~~
compilation terminated.
To get the best performance, it is recommended to adjust the number of threads by setting the environment variables OMP_NUM_THREADS, DP_INTRA_OP_PARALLELISM_THREADS, and DP_INTER_OP_PARALLELISM_THREADS. See https://deepmd.rtfd.io/parallelism/ for more information.
[2026-09-28 18:07:29,592] DEEPMD INFO    DeePMD version: 3.2.0
[2026-09-28 18:07:29,707] DEEPMD INFO    Set DP_TRITON_INFER to the desired level (0-3) before freezing; the selected Triton inference kernels are baked into the .pt2 archive.
[2026-09-28 18:07:30,454] DEEPMD INFO    Tracing the lower graph on CPU (make_fx)...
[2026-09-28 18:07:38,483] DEEPMD INFO    Exporting the traced graph (torch.export)...
[2026-09-28 18:07:53,617] DEEPMD INFO    Compiling the AOTInductor package for cuda:0 (the slowest freeze stage; typically several minutes)...
/tmp/tmpo81yk0bq/cuda_utils.c:1:10: fatal error: cuda.h: No such file or directory
    1 | #include "cuda.h"
      |          ^~~~~~~~
compilation terminated.
/tmp/tmp90_pyk_k/cuda_utils.c:1:10: fatal error: cuda.h: No such file or directory
    1 | #include "cuda.h"
      |          ^~~~~~~~
compilation terminated.
/tmp/tmpdsoghzeu/cuda_utils.c:1:10: fatal error: cuda.h: No such file or directory
    1 | #include "cuda.h"
      |          ^~~~~~~~
compilation terminated.
/tmp/tmp94v08ynk/cuda_utils.c:1:10: fatal error: cuda.h: No such file or directory
    1 | #include "cuda.h"
      |          ^~~~~~~~
compilation terminated.
/tmp/tmpup1pulgk/cuda_utils.c:1:10: fatal error: cuda.h: No such file or directory
    1 | #include "cuda.h"
      |          ^~~~~~~~
compilation terminated.
/tmp/tmpop58178f/cuda_utils.c:1:10: fatal error: cuda.h: No such file or directory
    1 | #include "cuda.h"
      |          ^~~~~~~~
compilation terminated.
/tmp/tmpwo0riq74/cuda_utils.c:1:10: fatal error: cuda.h: No such file or directory
    1 | #include "cuda.h"
      |          ^~~~~~~~
compilation terminated.
/tmp/tmp8hhnpogd/cuda_utils.c:1:10: fatal error: cuda.h: No such file or directory
    1 | #include "cuda.h"
      |          ^~~~~~~~
compilation terminated.
/tmp/tmpy8fifl5b/cuda_utils.c:1:10: fatal error: cuda.h: No such file or directory
    1 | #include "cuda.h"
      |          ^~~~~~~~
compilation terminated.
/tmp/tmpqxxsks77/cuda_utils.c:1:10: fatal error: cuda.h: No such file or directory
    1 | #include "cuda.h"
      |          ^~~~~~~~
compilation terminated.
/tmp/tmp3_v4a414/cuda_utils.c:1:10: fatal error: cuda.h: No such file or directory
    1 | #include "cuda.h"
      |          ^~~~~~~~
compilation terminated.
/tmp/tmphrjpqtv0/cuda_utils.c:1:10: fatal error: cuda.h: No such file or directory
    1 | #include "cuda.h"
      |          ^~~~~~~~
compilation terminated.
/tmp/tmp0qroo9wt/cuda_utils.c:1:10: fatal error: cuda.h: No such file or directory
    1 | #include "cuda.h"
      |          ^~~~~~~~
compilation terminated.
/tmp/tmp_d_z6j1k/cuda_utils.c:1:10: fatal error: cuda.h: No such file or directory
    1 | #include "cuda.h"
      |          ^~~~~~~~
compilation terminated.
/tmp/tmpkzwgr00u/cuda_utils.c:1:10: fatal error: cuda.h: No such file or directory
    1 | #include "cuda.h"
      |          ^~~~~~~~
compilation terminated.
/tmp/tmp30q69_ly/cuda_utils.c:1:10: fatal error: cuda.h: No such file or directory
    1 | #include "cuda.h"
      |          ^~~~~~~~
compilation terminated.
/tmp/tmpxttfz19m/cuda_utils.c:1:10: fatal error: cuda.h: No such file or directory
    1 | #include "cuda.h"
      |          ^~~~~~~~
compilation terminated.
/tmp/tmpel86l0vp/cuda_utils.c:1:10: fatal error: cuda.h: No such file or directory
    1 | #include "cuda.h"
      |          ^~~~~~~~
compilation terminated.
/tmp/tmpdbcksoqr/cuda_utils.c:1:10: fatal error: cuda.h: No such file or directory
    1 | #include "cuda.h"
      |          ^~~~~~~~
compilation terminated.
/tmp/tmpyjd7s046/cuda_utils.c:1:10: fatal error: cuda.h: No such file or directory
    1 | #include "cuda.h"
      |          ^~~~~~~~
compilation terminated.
/tmp/tmpm1iruo51/cuda_utils.c:1:10: fatal error: cuda.h: No such file or directory
    1 | #include "cuda.h"
      |          ^~~~~~~~
compilation terminated.
/tmp/tmp4ctd9i1x/cuda_utils.c:1:10: fatal error: cuda.h: No such file or directory
    1 | #include "cuda.h"
      |          ^~~~~~~~
compilation terminated.
/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/copyreg.py:99: FutureWarning: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead.
  return cls.__new__(cls, *args)
/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/torch/_inductor/compile_fx.py:320: UserWarning: TensorFloat32 tensor cores for float32 matrix multiplication available but not enabled. Consider setting `torch.set_float32_matmul_precision('high')` for better performance.
  warnings.warn(
W0928 18:08:40.499000 2911 site-packages/torch/_inductor/utils.py:1717] Not enough SMs to use max_autotune_gemm mode
/tmp/tmppguck85h/cuda_utils.c:1:10: fatal error: cuda.h: No such file or directory
    1 | #include "cuda.h"
      |          ^~~~~~~~
compilation terminated.
Traceback (most recent call last):
  File "/home/fengyu_xie/miniconda3/envs/deepmd4/bin/dp", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/deepmd/main.py", line 1158, in main
    deepmd_main(args)
  File "/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 367, in wrapper
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/deepmd/pt/entrypoints/main.py", line 694, in main
    freeze(model=FLAGS.model, output=FLAGS.output, head=FLAGS.head)
  File "/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/deepmd/pt/entrypoints/main.py", line 490, in freeze
    freeze_sezm_to_pt2(model, out_pt2, head=head)
  File "/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/deepmd/pt/entrypoints/freeze_pt2.py", line 961, in freeze_sezm_to_pt2
    aoti_compile_and_package(exported, package_path=out_path_str)
  File "/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/torch/_inductor/__init__.py", line 152, in aoti_compile_and_package
    return aot_inductor_minifier_wrapper(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/torch/_inductor/debug.py", line 1344, in aot_inductor_minifier_wrapper
    raise e
  File "/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/torch/_inductor/debug.py", line 1314, in aot_inductor_minifier_wrapper
    return func(
           ^^^^^
  File "/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/torch/_inductor/__init__.py", line 195, in _aoti_compile_and_package_inner
    aoti_files = aot_compile(gm, args, kwargs, options=inductor_configs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/torch/_inductor/__init__.py", line 311, in aot_compile
    return compile_fx_aot(
           ^^^^^^^^^^^^^^^
  File "/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/torch/_inductor/compile_fx.py", line 2086, in compile_fx_aot
    compiled_artifacts = compile_fx(
                         ^^^^^^^^^^^
  File "/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/torch/_inductor/compile_fx.py", line 2675, in compile_fx
    return compile_fx(
           ^^^^^^^^^^^
  File "/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/torch/_inductor/compile_fx.py", line 2721, in compile_fx
    return _maybe_wrap_and_compile_fx_main(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/torch/_inductor/compile_fx.py", line 2815, in _maybe_wrap_and_compile_fx_main
    return _compile_fx_main(
           ^^^^^^^^^^^^^^^^^
  File "/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/torch/_inductor/compile_fx.py", line 3001, in _compile_fx_main
    return inference_compiler(unlifted_gm, example_inputs_)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/torch/_functorch/_aot_autograd/schemas.py", line 1421, in __call__
    output_code = self.compiler_fn(gm, example_inputs)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/torch/_inductor/compile_fx.py", line 2876, in fw_compiler_base
    return compile_fx_forward(
           ^^^^^^^^^^^^^^^^^^^
  File "/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/torch/_inductor/compile_fx.py", line 2504, in compile_fx_forward
    result = inner_compile(
             ^^^^^^^^^^^^^^
  File "/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/contextlib.py", line 81, in inner
    return func(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^
  File "/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/torch/_inductor/compile_fx.py", line 827, in compile_fx_inner
    return wrap_compiler_debug(_compile_fx_inner, compiler_name="inductor")(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/torch/_dynamo/repro/after_aot.py", line 314, in debug_wrapper
    inner_compiled_fn = compiler_fn(gm, example_inputs)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/torch/_inductor/compile_fx.py", line 1038, in _compile_fx_inner
    raise InductorError(e, currentframe()).with_traceback(
  File "/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/torch/_inductor/compile_fx.py", line 1030, in _compile_fx_inner
    mb_compiled_graph = fx_codegen_and_compile(
                        ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/torch/_inductor/compile_fx.py", line 1836, in fx_codegen_and_compile
    return scheme.codegen_and_compile(gm, example_inputs, inputs_to_check, graph_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/torch/_inductor/compile_fx.py", line 1556, in codegen_and_compile
    wrapper_code, kernel_code = graph.codegen_with_cpp_wrapper()
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/torch/_inductor/graph.py", line 2506, in codegen_with_cpp_wrapper
    return self.codegen()
           ^^^^^^^^^^^^^^
  File "/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/torch/_inductor/graph.py", line 2555, in codegen
    self.scheduler.codegen()
  File "/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/torch/_inductor/scheduler.py", line 7352, in codegen
    else self._codegen(self.nodes)
         ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/torch/_inductor/scheduler.py", line 7641, in _codegen
    self.get_backend(device).codegen_node(node)
  File "/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/torch/_inductor/codegen/cuda_combined_scheduling.py", line 152, in codegen_node
    return self._triton_scheduling.codegen_node(node)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/torch/_inductor/codegen/simd.py", line 1864, in codegen_node
    return self._codegen_nodes(nodes, coalesce_analysis)  # type: ignore[arg-type]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/torch/_inductor/codegen/simd.py", line 1837, in _codegen_nodes
    return self.codegen_node_schedule(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/torch/_inductor/codegen/simd.py", line 1961, in codegen_node_schedule
    src_code = kernel.codegen_kernel()
               ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/torch/_inductor/codegen/triton.py", line 5760, in codegen_kernel
    **self.inductor_meta_common(),
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/torch/_inductor/codegen/triton.py", line 5568, in inductor_meta_common
    "backend_hash": torch.utils._triton.triton_hash_with_backend(),
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/torch/utils/_triton.py", line 219, in triton_hash_with_backend
    backend = triton_backend()
              ^^^^^^^^^^^^^^^^
  File "/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/torch/utils/_triton.py", line 193, in triton_backend
    target = driver.active.get_current_target()
             ^^^^^^^^^^^^^
  File "/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/triton/runtime/driver.py", line 39, in active
    self._active = self.default
                   ^^^^^^^^^^^^
  File "/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/triton/runtime/driver.py", line 33, in default
    self._default = _create_driver()
                    ^^^^^^^^^^^^^^^^
  File "/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/triton/runtime/driver.py", line 21, in _create_driver
    return active_drivers[0]()
           ^^^^^^^^^^^^^^^^^^^
  File "/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/triton/backends/nvidia/driver.py", line 336, in __init__
    self.utils = CudaUtils()  # TODO: make static
                 ^^^^^^^^^^^
  File "/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/triton/backends/nvidia/driver.py", line 66, in __init__
    mod = compile_module_from_src(
          ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/triton/runtime/build.py", line 115, in compile_module_from_src
    so = _build(name, src_path, tmpdir, library_dirs or [], include_dirs or [], libraries or [], ccflags or [])
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/triton/runtime/build.py", line 70, in _build
    subprocess.check_call(cc_cmd, stdout=subprocess.DEVNULL)
  File "/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/subprocess.py", line 413, in check_call
    raise CalledProcessError(retcode, cmd)
torch._inductor.exc.InductorError: CalledProcessError: Command '['x86_64-conda-linux-gnu-cc', '/tmp/tmppguck85h/cuda_utils.c', '-O3', '-shared', '-fPIC', '-Wno-psabi', '-o', '/tmp/tmppguck85h/cuda_utils.cpython-312-x86_64-linux-gnu.so', '-l:libcuda.so.1', '-L/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/triton/backends/nvidia/lib', '-L/usr/lib/wsl/lib', '-I/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/triton/backends/nvidia/include', '-I/tmp/tmppguck85h', '-I/home/fengyu_xie/miniconda3/envs/deepmd4/include/python3.12', '-I/home/fengyu_xie/miniconda3/envs/deepmd4/targets/x86_64-linux/include']' returned non-zero exit status 1.

Similar issue never happend with Bohrium image "registry.dp.tech/dptech/deepmd-kit:3.2.0-sm89" on 4090_24g nodes

My intial guess is the conda package provided in the conda-forge channel might not have been bundled correctly. Specifically, some issue might exist with the packaged CUDA.

Could you please check this issue?

DeePMD-kit Version

3.2.0

Backend and its version

pytorch==2.12.0

How did you download the software?

conda

Input Files, Running Commands, Error Log, etc.

Steps to Reproduce

• Install deepmd-kit 3.2.0 via default conda command as specified in the documentation
• dp --pt freeze -c model.ckpt.pt -o frozen

Further Information, Files, and Links

No response