[BUG] Honor data_stat_full for fine-tuning delta-bias statistics

Open
  • Is something wrong?
#6,101 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 3 days

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
48/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
python, pytorch
Domain
backend, machine-learning

Research direction

Read deepmd/pt/train/training.py around the change-by-statistic fine-tuning initialization and deepmd/pt/utils/stat.py around compute_output_stats and its full-scan fallback. Trace how residual predictions, frame filtering, type mapping, and the statistics cache are handled, then add the focused regression test described in the issue. Done means full mode matches an explicit all-frame least-squares reference for a rare element while sampled mode and scratch-training statistics retain their stated behavior.

Written by the indexing model from the issue text.

Description

bug
Bug summary

model.data_stat_full=true does not apply to the delta-bias recalibration performed by --finetune. The fine-tuning path uses change-by-statistic, but compute_output_stats disables the full-data scanner whenever model_forward is present and falls back to sampled statistics.

This can leave rare elements without an effective energy-reference correction, even when a shared HDF5 file already contains valid full-dataset absolute output statistics.

Requested interface: reuse the existing model.data_stat_full option for both ordinary output statistics and fine-tuning delta-bias statistics. Do not introduce a separate fine-tuning flag.

DeePMD-kit version

Observed with 3.2.1.dev45+gde3fc8f62.d20261021 on the outisli development branch with local modifications. The same fallback is also present in upstream master at a78bdfe8d5dc494a9d9a397e4a9e103a84205eaa (source links below).

Backend and installation

PyTorch 2.13.0+cu130; editable source installation on Linux. The observed case fine-tunes a spin-free DPA4 checkpoint into a native-spin model.

Input, command, and observed behavior

Relevant configuration excerpt, merged into an otherwise valid fine-tuning input:

{
  "model": {
    "data_stat_full": true,
    "data_stat_nbatch": 10
  },
  "training": {
    "stat_file": "all_399k.hdf5",
    "training_data": {
      "systems": "all_399k.lmdb",
      "batch_size": "mix:2000"
    }
  }
}

dp --pt train input.json --finetune pretrained.pt --skip-neighbor-stat

Startup emits:

Falling back to sampled output statistics: a full scan supports neither delta bias nor stats_distinguish_types=False.

In the observed MagPT case, the target collection contains 399,168 frames and 48 elements. Full absolute statistics were rebuilt successfully: the HDF5 records output_stat_full_scan=true and all 48 observed elements, and a scratch-training checkpoint loads those biases exactly. However, the sampled fine-tuning calibration leaves V and Ag near zero:

Element Full-data absolute bias (eV/atom) Inspected fine-tuned checkpoint bias (eV/atom)
V -2007.953713 approximately 0
Ag -4217.112746 0

That sampled fine-tuning run starts with a training energy RMSE of approximately 159 eV/atom. Enabling data_stat_full on the subsequent fine-tuning launch still produces the fallback warning above.

The absolute and fine-tuning biases are not expected to be identical: fine-tuning must account for the energy already predicted by the pretrained model. The problem is that requesting full-data recalibration still falls back to a sample that does not adequately cover the target elements.

Steps to reproduce
  1. Prepare a target dataset with rare elements that are not adequately represented in the default statistics sample, and a compatible pretrained checkpoint with different energy references.
  2. Generate valid full-dataset absolute output statistics in an HDF5 cache.
  3. Fine-tune the checkpoint on the target dataset with model.data_stat_full=true, pointing training.stat_file at that cache.
  4. Observe the fallback warning and inspect the recalibrated element biases rather than assuming that the HDF5 values were reused.
Cause

The fine-tuning initialization explicitly selects change-by-statistic:

The output-statistics implementation disables the full scanner for a model-dependent residual and bypasses the absolute-statistics cache:

Bypassing an absolute-label cache for model-dependent residual statistics is appropriate. Falling back to sampled data despite data_stat_full=true is the missing behavior.

Expected behavior and acceptance criteria
  • Reuse model.data_stat_full: when true, delta-bias recalibration visits every eligible target-training frame and fits the residual between the target labels and the current pretrained model predictions.
  • Accumulate residual-regression statistics in bounded-memory batches, retaining the existing type mapping, frame filtering, and bias-adjustment semantics.
  • Keep model-dependent residual statistics separate from absolute-label HDF5 statistics. Do not substitute cached absolute biases for the residual regression or overwrite the shared absolute-statistics cache with model-specific values.
  • Preserve the existing sampled behavior when data_stat_full=false, and retain the existing absolute-statistics path for training from scratch.
  • Add a focused regression test with an element absent from the initial sample but present later in the dataset. Full delta-bias calibration should match an explicit all-frame least-squares reference and produce the appropriate correction for that element.
Dominant language
Python
Stars
2.1k
Forks
663
Avg merge
3d 11h
Merged PRs (30d)
25

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from deepmodeling/deepmd-kit

All issues in deepmodeling/deepmd-kit

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.