docs(train): document stat_file, EMA shadow model, and LMDB datasets

Open
  • Is something wrong?
#5,925 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 3 days

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
55/100
Issue type
Documentation
Clarity
Clearly specified
Activity status
Quiet
Tech stack
numpy, python, pytorch, tensorflow
Domain
documentation

Research direction

Read the cited entries in deepmd/utils/argcheck.py, deepmd/tf/model/stat_file.py, deepmd/dpmodel/utils/lmdb_data.py, and deepmd/pt_expt/utils/lmdb_dataset.py, then inspect the data dispatch paths. Update doc/train/training-advanced.md, doc/data/system.md, and doc/train/training.md; run the specified grep checks, with completion shown by accurate defaults, backend restrictions, usage examples, and LMDB pointers.

Written by the indexing model from the issue text.

Description

Docs

Parent issue: #5917 (checklist items 10, 12, 16)

Scope

Three training-side features registered in the schema (deepmd/utils/argcheck.py) but lacking prose:

  1. stat_file / stat_file_mode (2b3bb05a7, feat(tf): support training stat_file; implementation deepmd/tf/model/stat_file.py; schema at argcheck.py:5300-5313,5348-5357): persist data statistics so later trainings skip recomputation. stat_file selects a .h5/.hdf5 HDF5 file or (otherwise) a directory of NumPy binaries; stat_file_mode is PyTorch-only (update = create/write missing stats; read = require an existing complete cache, read-only, safe to share one HDF5 file across training processes). Nothing in doc/train/ mentions them.
  2. EMA shadow model (57f870ff4, PyTorch): enable_ema, ema_decay, ema_ckpt_keep (argcheck.py:5264-5278,5425+). Only a passing ema_ckpt_keep mention exists inside the ckpt_keep_ratio description at doc/train/training-advanced.md:107.
  3. LMDB training datasets (565f4beec pt/dpmodel dataloader, 54f42d90c pt_expt, 9d6381602 max/filter mode, a3f548d7b full-validation lmdb): deepmd/dpmodel/utils/lmdb_data.py (pure Python lmdb/msgpack reader), deepmd/pt_expt/utils/lmdb_dataset.py. Not documented as a training data format in doc/data/ or doc/train/ (only as a dpa-adapt conversion target at doc/dpa_adapt/input_formats.md:105).

Implementation plan

1+2. doc/train/training-advanced.md
  • Add a "Data statistics cache (stat_file)" subsection:
    • Purpose (avoid re-running neighbor/statistics passes), stat_file path semantics (HDF5 vs npy directory by extension), backend support (TensorFlow and PyTorch — verify TF support from deepmd/tf/model/stat_file.py), and a minimal JSON example under the training section;
    • stat_file_mode (update default / read), PyTorch-only, parallel-training use case.
  • Add an "Exponential moving average (EMA) of parameters" subsection:
    • enable_ema (PyTorch only), ema_decay (document default from argcheck), the _ema-suffixed periodic checkpoints, ema_ckpt_keep retention, interaction with ckpt_keep_ratio; how to consume an EMA checkpoint (verify whether freeze/test can take it directly from deepmd/pt/entrypoints).
3. LMDB datasets — doc/data/system.md (+ pointer from doc/train/training.md)

Read deepmd/dpmodel/utils/lmdb_data.py, deepmd/pt_expt/utils/lmdb_dataset.py, and how training_data.systems entries are dispatched (e.g. deepmd/pt/data_utils/, deepmd/utils/data_system.py) before writing.

  • In doc/data/system.md, add LMDB to the supported data system formats:
    • which backends support it (PyTorch, PyTorch-Exportable, DP — verify; note TF does not if that is the case);
    • on-disk layout/metadata expectations as implemented (msgpack-serialized entries, metadata keys, type map storage — keep it concise and accurate);
    • how to point training_data.systems / validation_data.systems at an LMDB directory (verify path detection logic — suffix or directory marker);
    • how to produce LMDB datasets (link doc/dpa_adapt/input_formats.md conversion route).
  • In doc/train/training.md, add one sentence near the DataSystem summary paragraph noting LMDB systems are also accepted on supported backends (cross-ref the data docs).

Constraints

  • Verify defaults and backend support from code (argcheck entries and dataloader dispatch) — several of these are backend-restricted; state restrictions explicitly.
  • Keep JSON snippets minimal and valid (check-json runs in pre-commit only for repo files, not docs, but stay valid).

Verification

  • grep -n "stat_file" doc/train/training-advanced.md and grep -n -i "ema" doc/train/training-advanced.md show new subsections.
  • grep -n -i "lmdb" doc/data/system.md doc/train/training.md shows the new format docs.

Parent issue: #5917

Coding agent: opencode
opencode version: 1.18.9
Model: ustc/k3
Reasoning effort: max

Dominant language
Python
Stars
2.1k
Forks
663
Avg merge
3d 11h
Merged PRs (30d)
25

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from deepmodeling/deepmd-kit

All issues in deepmodeling/deepmd-kit

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.