Run-to-run non-determinism from floating-point atomics

オープン
#1,007 コメント 1 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 2 日以内に返信

まだ誰も着手していません。

評価

難易度
5/5
見積もり時間
1週間以上
初心者へのやさしさ
35/100
issue の種類
機能追加
明瞭さ
おおむね明確
活発さ
活発
技術スタック
python
領域
backend, testing

調査の方向性

Start with the fork’s DETERMINISM.md and gpu4pyscf/lib/tests/test_fixed_point.py to understand the proposed approach and existing checks. Review the listed kernel families and the unvalidated PBC, TDDFT, and direct-SCF Hessian paths to establish what an upstream change would need to cover. Done should include an agreed scope and reproducible tests showing the intended paths are bit-identical across runs.

索引モデルが issue の本文から書いたものです。

説明

Summary

Repeated runs of the same input on the same GPU give slightly different
energies, gradients and analytic Hessians. The dominant cause is the
double-precision atomicAdd reductions in the CUDA kernels, whose summation
order depends on thread scheduling. The spread is far below the DF, grid and
SCF convergence error, so this is a reproducibility problem, not an accuracy
one. It still rules out bit-for-bit regression testing and bisecting. Our
case: geometry optimizations of open- and closed-shell molecules (Sella,
gpu4pyscf forces). The optimizer amplifies the 2e-7 eV/Å force noise by
about 1.9x per step, so identical repeat optimizations ended in different
minima 23 % of the time (up to 23 kcal/mol apart), and we could not separate
optimizer bugs from floating-point noise until the forces were reproducible.

I have an opt-in fix working in a fork and would like to know whether a PR is
welcome, and in what form, before cleaning it up.

Claims are marked [measured] when a run backs them (see Evidence) and
[assumed] when inferred from code or documentation but not isolated in a
test.

Observed behaviour [measured]

Table 1. Run-to-run differences in stock gpu4pyscf.
Setup: DF wB97M-V/def2-SVPD + SMD · A100-SXM4 40 GB and 80 GB · CUDA 12.9 ·
gpu4pyscf 1.7.6, 1.8.1 and master (d0ea5b0) · 6 to 19 atom molecules with
H, C, N, O, F, P, S · fresh process per repeat.

Quantity Run-to-run difference Sample
SCF total energy median 3e-13 Ha, 22 % of pairs bit-identical, max 2e-12 Ha 192 repeat pairs from a production campaign (1.7.6); 48 same-GPU repeats; 20 repeats on master
Nuclear gradient RMS 2e-7 eV/Å (p90 9e-7), max component ~1e-6 eV/Å, 0 of 240 pairs identical same
Analytic Hessian 3e-8 to 7e-7 eV/Ų (5e-11 to 1.5e-9 Ha/Bohr²), 0 of 13 molecules identical 10 molecules x 2 repeats + 3 molecules x 3 repeats, master

Same-process repeats on one GPU and repeats across the four GPUs of a node
give the same spread (|dE| <= 9e-13 Ha, force RMS 1 to 2e-7 eV/Å), so this is
not a device-to-device effect. The energy is occasionally identical because
the XC quadrature runs through cuBLAS/cuTENSOR, which are deterministic; the
J/K, DF-gradient and solvent kernels, which use atomics, agreed in 0 of 21 to
28 repeats in every benchmark.

CPU PySCF has the same class of problem [measured] (Table 2, last row).
Single-threaded runs, or OpenMP at 1 thread with OpenBLAS at 8, are
bit-identical; OpenMP at 8 with OpenBLAS at 1 is not. The source is
PySCF's own OpenMP code: per-thread partial sums merged under
#pragma omp critical in arrival order, e.g. VXCdot_ao_ao in
lib/dft/nr_numint.c (XC matrix, 1e-15 on a fixed density) and the K-split
NPdgemm behind lib.dot in lib/np_helper/npdot.c (DF J/K, 1e-15). On
CPU an ordered merge of the thread buffers would fix it, and PySCF already
has one (NPomp_dsum_reduce_inplace). This issue is about the GPU path,
where thousands of concurrent adders make ordered merging impractical and
fixed point is the natural tool.

Not measured: translational/rotational invariance of the gradient beyond
this noise level.

Reproducer

import sys, numpy as np, pyscf
from gpu4pyscf import dft
mol = pyscf.M(atom="O 0 0 0; H 0 0.76 0.59; H 0 -0.76 0.59; F 1.8 0 0.2; S 1.2 1.5 -0.4",
              basis="def2-svp", spin=1, verbose=0)
mf = dft.UKS(mol, xc="wb97x").density_fit()
mf.grids.level = 1
mf.conv_tol = 1e-10
mf.kernel()
g = mf.nuc_grad_method().kernel()
h = mf.Hessian().kernel()
np.savez(sys.argv[1], e=mf.e_tot, g=g, h=h)

Run as separate processes and compare the .npz files.

Table 2. Reproducer results.
Setup: 3 runs per row, separate processes · shared login-node
A100-PCIE-40GB · driver 580.178.04 · CUDA runtime 12.9 · PySCF 2.14.0 ·
CuPy 14.1.1 · CPU row: 8 OpenMP threads · values are max |difference|
between runs · largest gradient component 0.21 Ha/Bohr, largest Hessian
element 1.03 Ha/Bohr².

build max ΔE (Ha) max Δgradient (Ha/Bohr) max ΔHessian (Ha/Bohr²)
gpu4pyscf-cuda12x 1.7.6 wheel 0 (3/3 identical) 7e-15 to 1.8e-14 0 (3/3 identical)
fork, upstream 1.8 base (d0ea5b0), deterministic mode off 1.1e-13 4.9e-10 to 9.8e-10 1.4e-10 to 1.5e-10
fork, GPU4PYSCF_DETERMINISTIC=1 0 (3/3 identical) 0 (3/3 identical) 0 (3/3 identical)
fork, deterministic vs non-deterministic (same build) 1.1e-12 5.0e-10 2.3e-10
PySCF 2.14.0 on CPU, 8 threads 1.1e-13 to 6.8e-13 1.9e-8 to 2.3e-8 3.5e-9 to 3.8e-9

Two cautions. The reproducer is small and gas-phase; on the 1.7.6 wheel only
the gradient moved. The spread grows with system size and with the number of
atomic kernels on the path (SMD, DF gradient), hence the larger numbers in
Table 1. And three runs is a small sample: in production the J/K,
DF-gradient and solvent kernels never agreed in any of 28 repeats, so two
identical runs do not disprove the effect.

Where the non-determinism comes from

Demonstrated sources [measured]
  1. atomicAdd(double*, double) in the integral kernels. Controlled A/B:
    same source, same compiler, double atomics (dbl) vs int64 fixed-point
    atomics (fp), nothing else changed. Kernels with plain-array inputs go
    from "no two repeats agree" to "all agree" (Table E1). Libraries and
    kernels, from the fork's diff:
    • libgvhf_rys: J/K contraction and the ip1 / ip2 derivative kernels
      (rys_contract_jk*.cu, unrolled_rys_jk*.cu, unrolled_ejk_ip*.cu,
      ejk_int3c2e_ip*.cu, e_int2c2e_ip2.cu); validated
    • libgvhf_md: J engine (md_contract_j.cu, unrolled_md_j*.cu);
      validated via the DF J path on master
    • libgint: int3c1e* and int1e_grids; validated via the SMD
      potential and its charge gradient
    • libpbc: rys_contract_j/k, unrolled_rys_k.cu, e_int2c2e_ip1.cu;
      converted because they share dot_dm with the Rys kernels, NOT
      validated (we run no PBC)
  2. Shared-memory race in _screen_index (lib/gdft/nr_eval_gto.cu).
    A freshly built Grids object keeps or drops a borderline shell at random,
    so the XC matrix differs at cutoff level. Upstream issue #919, fixed in
    PR #925 after 1.8.1. Confirmed by comparing the screening mask of
    repeatedly built Grids objects: differs on 1.8.1, identical 21/21 on
    master.
  3. Duplicate AO pairs in the packed DF tensor. On master the packed
    cderi stores some AO pairs as both ij and ji (224 of 18,290 for a
    12-atom molecule) with values differing at 1e-13, and
    decompress_and_fill lets the last block win. With every kernel already
    integer-accumulated this alone left the SCF energy varying at <= 1.8e-12
    Ha (10/21 identical); a first-occurrence write mask gave 21/21.
Likely sources, not isolated [assumed]
  1. cupy.add.at scatters in gpu4pyscf/hessian/{rhf,rks,uks}.py. CuPy
    implements these with float atomics. We replaced all 14 calls with a
    fixed-order one-hot matrix product and the Hessian became bit-identical,
    but we never tested the Hessian with only cupy.add.at left in place, so
    its own contribution is unknown.
  2. Block and batch sizes derived from current free memory
    (cupy_helper avail-mem queries), which make the reduction grouping
    depend on the memory-pool state. Replaced by a fixed fraction of total
    memory. Before the screening race was fixed this moved the SCF energy from
    18/28 to 22/28 identical and the DF gradient from 17/28 to 19/28; after
    the fix both settings reach 28/28, so the effect was never isolated.

Proposed fix (implemented in fork)

Fork: https://github.com/oliverhvidsten/gpu4pyscf (branch master, tag
v1.8.1-det.1, on upstream d0ea5b0). Write-up:
https://github.com/oliverhvidsten/gpu4pyscf/blob/master/DETERMINISM.md.
Tests: gpu4pyscf/lib/tests/test_fixed_point.py (7 tests, including a
two-process bit-identity check).

Opt-in via GPU4PYSCF_DETERMINISTIC=1; default behaviour unchanged.

  1. Fixed-point accumulation (AMBER SPFP scheme). lib/fixed_point.cuh
    defines fp_atomic_add(double*, double). With -DFIXED_POINT_ACCUM it
    scales by 2^48, rounds to int64 and does an integer atomicAdd into the
    same 8-byte slot; integer addition is associative, so the result is
    order-independent. Without the define it is the plain double atomicAdd.
    The four libraries are built a second time as lib*_fp.so, and the
    Python call sites convert the buffer back once with from_fixed().
  2. Deterministic blocking. Block sizes from a fixed fraction of total
    memory; the cderi unpack writes each AO pair once; cupy.add.at
    replaced by a one-hot matrix product (lib/fixed_point.add_at) with a
    fixed summation order.

Results [measured, Tables E1 to E5]:

  • Bit-identical across repeats: SCF energy, density, orbital energies,
    forces and analytic Hessian, every repeat of every molecule (28/28, 21/21
    and 10/10 in three independent runs).
  • Across nodes of the same GPU model: one data point (H2O Hessian, two A100
    nodes). [assumed beyond that]
  • Agreement with the stock build: 1e-12 Ha in the SCF energy, 4e-13 Ha/Bohr
    for a single gradient kernel on identical inputs, 3e-11 to 1.4e-9 Ha/Bohr²
    in the Hessian; all within the stock build's own spread.
  • Wall time, deterministic/stock: 0.99 to 1.02 for SCF and gradient, 1.00 to
    1.05 for SCF + Hessian; one 0.1 s piece (SMD potential) at 1.07.

Validated: molecular DF-DFT (RKS/UKS) with SMD, SCF, gradient and analytic
Hessian, on H, C, N, O, F, P, S. PBC, TDDFT and direct-SCF Hessian paths have
the fixed-point kernels but are unvalidated.

Evidence

All runs on Perlmutter A100 nodes with fresh SCFs. "n/28" = n of 28
(7 molecules x 4 repeats) repeat values bit-identical to the first repeat of
their molecule; 7/28 means no two repeats ever agreed.

Table E1. Controlled A/B: bit-identical repeats per piece.
Setup: same source (1.7.6 kernels + the #925 screening fix) and same
compiler for dbl, fpnb, fp · A100-SXM4-80GB · 7 molecules x 4 repeats ·
one process per variant · Slurm 59791528.

piece stock (released wheel) dbl (rebuilt, double atomics) fpnb (fixed point only) fp (fixed point + deterministic blocking)
direct J/K (RYS_build_jk) 7/28 7/28 28/28 28/28
XC potential matrix 15/28 16/28 28/28 28/28
SMD potential (int1e_grids) 7/28 7/28 28/28 28/28
DF 2e gradient (ejk_int3c2e_ip1 + e_int2c2e_ip1) 7/28 7/28 28/28 28/28
SMD charge gradient (int1e_grids_ip1) 7/28 7/28 28/28 28/28
converged SCF energy 12/28 16/28 28/28 28/28
converged density 7/28 7/28 28/28 28/28
total nuclear gradient 7/28 7/28 28/28 28/28
direct per-atom 2e gradient (RYS_per_atom_jk_ip1, 1.7.6 version) 7/28 7/28 25/28 25/28

Master rewrote the last kernel; its master version is bit-identical under
fixed point in our kernel-family test.

Table E2. Wall-time cost of fixed point.
Setup: same job as E1 · fp vs dbl, so only the accumulation differs · median
and range over the 7 molecules · mean of 4 repeats per molecule.

piece median fp/dbl range
SCF 1.014 0.993 to 1.022
gradient 1.004 0.995 to 1.020
DF 2e gradient 0.988 0.984 to 1.007
direct J/K 1.035 0.772 to 1.084
SMD potential (0.1 s absolute) 1.070 0.994 to 1.094

Table E3. Accuracy of fixed point against double.
Setup: same job as E1 · repeat-0 values · max abs difference over the 7
molecules.

quantity max abs(fp - dbl) max abs(dbl - stock)
SCF energy 1.8e-12 Ha 9.1e-13 Ha
direct 2e gradient, identical input density 4.2e-13 Ha/Bohr 5.7e-14 Ha/Bohr
total gradient, each from its own converged SCF 5.6e-8 Ha/Bohr 6.4e-8 Ha/Bohr

The 5e-8 Ha/Bohr total-gradient difference is SCF-state noise carried into
the gradient, not fixed-point rounding: the same kernel on identical inputs
agrees to 4e-13, and fp vs dbl is no larger than dbl vs stock.

Table E4. Full fork patch on upstream master.
Setup: master d0ea5b0 + fork patch · 7 molecules x 3 repeats · production
SCF + forces · login-node A100 · 34 recorded pieces: DF tensor, DF J/K,
direct J/K, direct per-atom gradient, XC, VV10, SMD potential, SCF energy,
density, orbital energies, DF and SMD gradient terms, total forces.

quantity stock master, identical repeats deterministic, identical repeats
SCF energy 10/21 (spread <= 1.8e-12 Ha) 21/21
converged density 7/21 21/21
all 34 pieces incl. total forces not all 21/21 each

Table E5. Analytic Hessians, 10 production molecules.
Setup: master + fork patch · production settings (DF, SMD, level-4 grid) ·
molecules with S/F/N/P, 6 to 15 atoms, 136 to 355 AOs, charges -2 to +2,
singlets and doublets · 2 fresh SCF + Hessian repeats per molecule and mode ·
one A100 per (molecule, mode) · Slurm 60338207 · dH = max |difference|
between the two repeats.

molecule natm nao stock: E identical / H identical / max dH (eV/Ų) det: E identical / H identical / max dH det vs stock, max dH (eV/Ų) max abs(H) (eV/Ų) s per repeat, stock / det
frag_0 9 212 no / no / 9.5e-08 yes / yes / 0 1.0e-07 115.8 22 / 23
frag_51 6 139 no / no / 7.1e-07 yes / yes / 0 9.1e-07 56.6 20 / 21
recomb_746 13 305 no / no / 1.3e-07 yes / yes / 0 1.6e-07 119.4 42 / 42
recomb_2905 6 139 no / no / 5.3e-08 yes / yes / 0 3.2e-07 99.3 22 / 22
recomb_6180 6 139 no / no / 3.9e-08 yes / yes / 0 5.5e-08 73.1 22 / 22
recomb_6573 12 226 no / no / 2.9e-07 yes / yes / 0 3.9e-07 191.1 44 / 44
recomb_13108 15 355 no / no / 3.8e-07 yes / yes / 0 2.2e-07 89.8 55 / 56
recomb_17591 6 136 no / no / 2.9e-08 yes / yes / 0 8.8e-08 80.8 21 / 22
recomb_17788 10 173 no / no / 4.7e-08 yes / yes / 0 6.3e-08 63.3 32 / 33
recomb_18904 13 231 yes / no / 1.9e-07 yes / yes / 0 2.2e-07 147.1 50 / 51
total E 1/10, H 0/10 E 10/10, H 10/10 all < 1e-6

Table E6. Analytic Hessians, three small test molecules.
Setup: master + fork patch · DF, SMD, level-1 grid · 3 fresh SCF + Hessian
repeats per mode · A100 · Slurm 60179426 · spread = max |difference| across
repeats.

molecule stock Hessian spread (Ha/Bohr²) deterministic spread det vs stock (Ha/Bohr²) note
H2O (RKS) 5.0e-11 0 3.5e-11 also bit-identical on a second node
HOFS radical (UKS) 2.4e-10 0 2.5e-10
C2H5OF triplet (UKS) 1.5e-9 0 1.4e-9

Known limitations, for discussion

  • Accumulator range [measured on S and P, assumed beyond]. A 48-bit
    shift holds ±2^15 per accumulator. A single term above that saturates and
    a sum beyond 2^63 wraps, both silently. The nuclear-attraction Hessian
    terms (ejk_int3c2e_ip2 with nuclei as point charges) scale roughly as
    Z·ρ(0): for sulfur the self-block diagonal came out 32998 Ha/Bohr² too
    large; O and F were in range. The fork guards that kernel by scaling the
    point charges down by a power of two chosen from Z_max
    (charge_scale_for_range, 16 for P/S) and the result back up. A headroom
    probe shows S needs >= 2, so the margin is 8x, and Table E5 exercises the
    guard on every S/P case. It is calibrated on S and extrapolated as Z⁴;
    heavier elements (Cl, Br, I, metals) and the other converted Hessian
    kernels are unchecked. Before a PR I would add a device-side overflow flag
    that raises instead of returning a silently wrong number. A smaller shift
    (e.g. 40) would widen the range at the cost of resolution; input on that
    trade-off, or on a per-kernel shift, is welcome.
  • Diff size. About 12k CUDA lines across ~45 .cu files, almost all the
    mechanical atomicAdd → fp_atomic_add substitution in the unrolled
    kernels; ~470 Python lines across 26 files. Alternatives that leave the
    kernels alone (deterministic tree reductions) would need per-kernel
    restructuring and did not look cheaper.
  • Distribution. The _fp libraries roughly double the compiled-library
    size, so a build option may suit better than the default wheel. Either
    way works for us.
  • Scope of validation. One GPU architecture (A100), one CUDA toolkit
    (12.9), molecular DF-DFT only. We expect bit-identity only within one GPU
    model / driver / cuBLAS version, since the contraction libraries pick
    kernels per architecture. [assumed]

Questions

  1. Is an opt-in deterministic mode something you would accept upstream?
  2. If so, as a build-time option (-DBUILD_FIXED_POINT=ON) producing
    parallel _fp libraries, as in the fork, or another mechanism?
  3. Any preference on the range/overflow handling above?
主要言語
Cuda
スター
371
フォーク
108
平均マージ
2日 6時間
マージ済み PR(30日)
51

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

pyscf/gpu4pyscf のほかの issue

pyscf/gpu4pyscf の issue をすべて見る

似ている issue

Backend & API Design の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。