[BUG] Honor data_stat_full for fine-tuning delta-bias statistics

オープン
  • 何か問題がありますか？
#6,101 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 3 日以内に返信

まだ誰も着手していません。

評価

難易度
4/5
見積もり時間
3〜5日
初心者へのやさしさ
48/100
issue の種類
バグ
明瞭さ
明確に書かれている
活発さ
活発
技術スタック
python, pytorch
領域
backend, machine-learning

調査の方向性

Read deepmd/pt/train/training.py around the change-by-statistic fine-tuning initialization and deepmd/pt/utils/stat.py around compute_output_stats and its full-scan fallback. Trace how residual predictions, frame filtering, type mapping, and the statistics cache are handled, then add the focused regression test described in the issue. Done means full mode matches an explicit all-frame least-squares reference for a rare element while sampled mode and scratch-training statistics retain their stated behavior.

索引モデルが issue の本文から書いたものです。

説明

bug
Bug summary

model.data_stat_full=true does not apply to the delta-bias recalibration performed by --finetune. The fine-tuning path uses change-by-statistic, but compute_output_stats disables the full-data scanner whenever model_forward is present and falls back to sampled statistics.

This can leave rare elements without an effective energy-reference correction, even when a shared HDF5 file already contains valid full-dataset absolute output statistics.

Requested interface: reuse the existing model.data_stat_full option for both ordinary output statistics and fine-tuning delta-bias statistics. Do not introduce a separate fine-tuning flag.

DeePMD-kit version

Observed with 3.2.1.dev45+gde3fc8f62.d20261021 on the outisli development branch with local modifications. The same fallback is also present in upstream master at a78bdfe8d5dc494a9d9a397e4a9e103a84205eaa (source links below).

Backend and installation

PyTorch 2.13.0+cu130; editable source installation on Linux. The observed case fine-tunes a spin-free DPA4 checkpoint into a native-spin model.

Input, command, and observed behavior

Relevant configuration excerpt, merged into an otherwise valid fine-tuning input:

{
  "model": {
    "data_stat_full": true,
    "data_stat_nbatch": 10
  },
  "training": {
    "stat_file": "all_399k.hdf5",
    "training_data": {
      "systems": "all_399k.lmdb",
      "batch_size": "mix:2000"
    }
  }
}

dp --pt train input.json --finetune pretrained.pt --skip-neighbor-stat

Startup emits:

Falling back to sampled output statistics: a full scan supports neither delta bias nor stats_distinguish_types=False.

In the observed MagPT case, the target collection contains 399,168 frames and 48 elements. Full absolute statistics were rebuilt successfully: the HDF5 records output_stat_full_scan=true and all 48 observed elements, and a scratch-training checkpoint loads those biases exactly. However, the sampled fine-tuning calibration leaves V and Ag near zero:

Element Full-data absolute bias (eV/atom) Inspected fine-tuned checkpoint bias (eV/atom)
V -2007.953713 approximately 0
Ag -4217.112746 0

That sampled fine-tuning run starts with a training energy RMSE of approximately 159 eV/atom. Enabling data_stat_full on the subsequent fine-tuning launch still produces the fallback warning above.

The absolute and fine-tuning biases are not expected to be identical: fine-tuning must account for the energy already predicted by the pretrained model. The problem is that requesting full-data recalibration still falls back to a sample that does not adequately cover the target elements.

Steps to reproduce
  1. Prepare a target dataset with rare elements that are not adequately represented in the default statistics sample, and a compatible pretrained checkpoint with different energy references.
  2. Generate valid full-dataset absolute output statistics in an HDF5 cache.
  3. Fine-tune the checkpoint on the target dataset with model.data_stat_full=true, pointing training.stat_file at that cache.
  4. Observe the fallback warning and inspect the recalibrated element biases rather than assuming that the HDF5 values were reused.
Cause

The fine-tuning initialization explicitly selects change-by-statistic:

The output-statistics implementation disables the full scanner for a model-dependent residual and bypasses the absolute-statistics cache:

Bypassing an absolute-label cache for model-dependent residual statistics is appropriate. Falling back to sampled data despite data_stat_full=true is the missing behavior.

Expected behavior and acceptance criteria
  • Reuse model.data_stat_full: when true, delta-bias recalibration visits every eligible target-training frame and fits the residual between the target labels and the current pretrained model predictions.
  • Accumulate residual-regression statistics in bounded-memory batches, retaining the existing type mapping, frame filtering, and bias-adjustment semantics.
  • Keep model-dependent residual statistics separate from absolute-label HDF5 statistics. Do not substitute cached absolute biases for the residual regression or overwrite the shared absolute-statistics cache with model-specific values.
  • Preserve the existing sampled behavior when data_stat_full=false, and retain the existing absolute-statistics path for training from scratch.
  • Add a focused regression test with an element absent from the initial sample but present later in the dataset. Full delta-bias calibration should match an explicit all-frame least-squares reference and produce the appropriate correction for that element.
主要言語
Python
スター
2.1k
フォーク
663
平均マージ
3日 11時間
マージ済み PR（30日）
25

環境構築

Codespaces で開く

このプロジェクトの開発コンテナを、あなたの GitHub アカウントでブラウザ上に起動します。

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

deepmodeling/deepmd-kit のほかの issue

deepmodeling/deepmd-kit の issue をすべて見る

似ている issue

Python の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。