Run-to-run non-determinism from floating-point atomics

Aperta
#1,007 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 2 giorni

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
35/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
python
Ambito
backend, testing

Direzione di ricerca

Start with the fork’s DETERMINISM.md and gpu4pyscf/lib/tests/test_fixed_point.py to understand the proposed approach and existing checks. Review the listed kernel families and the unvalidated PBC, TDDFT, and direct-SCF Hessian paths to establish what an upstream change would need to cover. Done should include an agreed scope and reproducible tests showing the intended paths are bit-identical across runs.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Summary

Repeated runs of the same input on the same GPU give slightly different
energies, gradients and analytic Hessians. The dominant cause is the
double-precision atomicAdd reductions in the CUDA kernels, whose summation
order depends on thread scheduling. The spread is far below the DF, grid and
SCF convergence error, so this is a reproducibility problem, not an accuracy
one. It still rules out bit-for-bit regression testing and bisecting. Our
case: geometry optimizations of open- and closed-shell molecules (Sella,
gpu4pyscf forces). The optimizer amplifies the 2e-7 eV/Å force noise by
about 1.9x per step, so identical repeat optimizations ended in different
minima 23 % of the time (up to 23 kcal/mol apart), and we could not separate
optimizer bugs from floating-point noise until the forces were reproducible.

I have an opt-in fix working in a fork and would like to know whether a PR is
welcome, and in what form, before cleaning it up.

Claims are marked [measured] when a run backs them (see Evidence) and
[assumed] when inferred from code or documentation but not isolated in a
test.

Observed behaviour [measured]

Table 1. Run-to-run differences in stock gpu4pyscf.
Setup: DF wB97M-V/def2-SVPD + SMD · A100-SXM4 40 GB and 80 GB · CUDA 12.9 ·
gpu4pyscf 1.7.6, 1.8.1 and master (d0ea5b0) · 6 to 19 atom molecules with
H, C, N, O, F, P, S · fresh process per repeat.

Quantity Run-to-run difference Sample
SCF total energy median 3e-13 Ha, 22 % of pairs bit-identical, max 2e-12 Ha 192 repeat pairs from a production campaign (1.7.6); 48 same-GPU repeats; 20 repeats on master
Nuclear gradient RMS 2e-7 eV/Å (p90 9e-7), max component ~1e-6 eV/Å, 0 of 240 pairs identical same
Analytic Hessian 3e-8 to 7e-7 eV/Ų (5e-11 to 1.5e-9 Ha/Bohr²), 0 of 13 molecules identical 10 molecules x 2 repeats + 3 molecules x 3 repeats, master

Same-process repeats on one GPU and repeats across the four GPUs of a node
give the same spread (|dE| <= 9e-13 Ha, force RMS 1 to 2e-7 eV/Å), so this is
not a device-to-device effect. The energy is occasionally identical because
the XC quadrature runs through cuBLAS/cuTENSOR, which are deterministic; the
J/K, DF-gradient and solvent kernels, which use atomics, agreed in 0 of 21 to
28 repeats in every benchmark.

CPU PySCF has the same class of problem [measured] (Table 2, last row).
Single-threaded runs, or OpenMP at 1 thread with OpenBLAS at 8, are
bit-identical; OpenMP at 8 with OpenBLAS at 1 is not. The source is
PySCF's own OpenMP code: per-thread partial sums merged under
#pragma omp critical in arrival order, e.g. VXCdot_ao_ao in
lib/dft/nr_numint.c (XC matrix, 1e-15 on a fixed density) and the K-split
NPdgemm behind lib.dot in lib/np_helper/npdot.c (DF J/K, 1e-15). On
CPU an ordered merge of the thread buffers would fix it, and PySCF already
has one (NPomp_dsum_reduce_inplace). This issue is about the GPU path,
where thousands of concurrent adders make ordered merging impractical and
fixed point is the natural tool.

Not measured: translational/rotational invariance of the gradient beyond
this noise level.

Reproducer

import sys, numpy as np, pyscf
from gpu4pyscf import dft
mol = pyscf.M(atom="O 0 0 0; H 0 0.76 0.59; H 0 -0.76 0.59; F 1.8 0 0.2; S 1.2 1.5 -0.4",
              basis="def2-svp", spin=1, verbose=0)
mf = dft.UKS(mol, xc="wb97x").density_fit()
mf.grids.level = 1
mf.conv_tol = 1e-10
mf.kernel()
g = mf.nuc_grad_method().kernel()
h = mf.Hessian().kernel()
np.savez(sys.argv[1], e=mf.e_tot, g=g, h=h)

Run as separate processes and compare the .npz files.

Table 2. Reproducer results.
Setup: 3 runs per row, separate processes · shared login-node
A100-PCIE-40GB · driver 580.178.04 · CUDA runtime 12.9 · PySCF 2.14.0 ·
CuPy 14.1.1 · CPU row: 8 OpenMP threads · values are max |difference|
between runs · largest gradient component 0.21 Ha/Bohr, largest Hessian
element 1.03 Ha/Bohr².

build max ΔE (Ha) max Δgradient (Ha/Bohr) max ΔHessian (Ha/Bohr²)
gpu4pyscf-cuda12x 1.7.6 wheel 0 (3/3 identical) 7e-15 to 1.8e-14 0 (3/3 identical)
fork, upstream 1.8 base (d0ea5b0), deterministic mode off 1.1e-13 4.9e-10 to 9.8e-10 1.4e-10 to 1.5e-10
fork, GPU4PYSCF_DETERMINISTIC=1 0 (3/3 identical) 0 (3/3 identical) 0 (3/3 identical)
fork, deterministic vs non-deterministic (same build) 1.1e-12 5.0e-10 2.3e-10
PySCF 2.14.0 on CPU, 8 threads 1.1e-13 to 6.8e-13 1.9e-8 to 2.3e-8 3.5e-9 to 3.8e-9

Two cautions. The reproducer is small and gas-phase; on the 1.7.6 wheel only
the gradient moved. The spread grows with system size and with the number of
atomic kernels on the path (SMD, DF gradient), hence the larger numbers in
Table 1. And three runs is a small sample: in production the J/K,
DF-gradient and solvent kernels never agreed in any of 28 repeats, so two
identical runs do not disprove the effect.

Where the non-determinism comes from

Demonstrated sources [measured]
  1. atomicAdd(double*, double) in the integral kernels. Controlled A/B:
    same source, same compiler, double atomics (dbl) vs int64 fixed-point
    atomics (fp), nothing else changed. Kernels with plain-array inputs go
    from "no two repeats agree" to "all agree" (Table E1). Libraries and
    kernels, from the fork's diff:
    • libgvhf_rys: J/K contraction and the ip1 / ip2 derivative kernels
      (rys_contract_jk*.cu, unrolled_rys_jk*.cu, unrolled_ejk_ip*.cu,
      ejk_int3c2e_ip*.cu, e_int2c2e_ip2.cu); validated
    • libgvhf_md: J engine (md_contract_j.cu, unrolled_md_j*.cu);
      validated via the DF J path on master
    • libgint: int3c1e* and int1e_grids; validated via the SMD
      potential and its charge gradient
    • libpbc: rys_contract_j/k, unrolled_rys_k.cu, e_int2c2e_ip1.cu;
      converted because they share dot_dm with the Rys kernels, NOT
      validated (we run no PBC)
  2. Shared-memory race in _screen_index (lib/gdft/nr_eval_gto.cu).
    A freshly built Grids object keeps or drops a borderline shell at random,
    so the XC matrix differs at cutoff level. Upstream issue #919, fixed in
    PR #925 after 1.8.1. Confirmed by comparing the screening mask of
    repeatedly built Grids objects: differs on 1.8.1, identical 21/21 on
    master.
  3. Duplicate AO pairs in the packed DF tensor. On master the packed
    cderi stores some AO pairs as both ij and ji (224 of 18,290 for a
    12-atom molecule) with values differing at 1e-13, and
    decompress_and_fill lets the last block win. With every kernel already
    integer-accumulated this alone left the SCF energy varying at <= 1.8e-12
    Ha (10/21 identical); a first-occurrence write mask gave 21/21.
Likely sources, not isolated [assumed]
  1. cupy.add.at scatters in gpu4pyscf/hessian/{rhf,rks,uks}.py. CuPy
    implements these with float atomics. We replaced all 14 calls with a
    fixed-order one-hot matrix product and the Hessian became bit-identical,
    but we never tested the Hessian with only cupy.add.at left in place, so
    its own contribution is unknown.
  2. Block and batch sizes derived from current free memory
    (cupy_helper avail-mem queries), which make the reduction grouping
    depend on the memory-pool state. Replaced by a fixed fraction of total
    memory. Before the screening race was fixed this moved the SCF energy from
    18/28 to 22/28 identical and the DF gradient from 17/28 to 19/28; after
    the fix both settings reach 28/28, so the effect was never isolated.

Proposed fix (implemented in fork)

Fork: https://github.com/oliverhvidsten/gpu4pyscf (branch master, tag
v1.8.1-det.1, on upstream d0ea5b0). Write-up:
https://github.com/oliverhvidsten/gpu4pyscf/blob/master/DETERMINISM.md.
Tests: gpu4pyscf/lib/tests/test_fixed_point.py (7 tests, including a
two-process bit-identity check).

Opt-in via GPU4PYSCF_DETERMINISTIC=1; default behaviour unchanged.

  1. Fixed-point accumulation (AMBER SPFP scheme). lib/fixed_point.cuh
    defines fp_atomic_add(double*, double). With -DFIXED_POINT_ACCUM it
    scales by 2^48, rounds to int64 and does an integer atomicAdd into the
    same 8-byte slot; integer addition is associative, so the result is
    order-independent. Without the define it is the plain double atomicAdd.
    The four libraries are built a second time as lib*_fp.so, and the
    Python call sites convert the buffer back once with from_fixed().
  2. Deterministic blocking. Block sizes from a fixed fraction of total
    memory; the cderi unpack writes each AO pair once; cupy.add.at
    replaced by a one-hot matrix product (lib/fixed_point.add_at) with a
    fixed summation order.

Results [measured, Tables E1 to E5]:

  • Bit-identical across repeats: SCF energy, density, orbital energies,
    forces and analytic Hessian, every repeat of every molecule (28/28, 21/21
    and 10/10 in three independent runs).
  • Across nodes of the same GPU model: one data point (H2O Hessian, two A100
    nodes). [assumed beyond that]
  • Agreement with the stock build: 1e-12 Ha in the SCF energy, 4e-13 Ha/Bohr
    for a single gradient kernel on identical inputs, 3e-11 to 1.4e-9 Ha/Bohr²
    in the Hessian; all within the stock build's own spread.
  • Wall time, deterministic/stock: 0.99 to 1.02 for SCF and gradient, 1.00 to
    1.05 for SCF + Hessian; one 0.1 s piece (SMD potential) at 1.07.

Validated: molecular DF-DFT (RKS/UKS) with SMD, SCF, gradient and analytic
Hessian, on H, C, N, O, F, P, S. PBC, TDDFT and direct-SCF Hessian paths have
the fixed-point kernels but are unvalidated.

Evidence

All runs on Perlmutter A100 nodes with fresh SCFs. "n/28" = n of 28
(7 molecules x 4 repeats) repeat values bit-identical to the first repeat of
their molecule; 7/28 means no two repeats ever agreed.

Table E1. Controlled A/B: bit-identical repeats per piece.
Setup: same source (1.7.6 kernels + the #925 screening fix) and same
compiler for dbl, fpnb, fp · A100-SXM4-80GB · 7 molecules x 4 repeats ·
one process per variant · Slurm 59791528.

piece stock (released wheel) dbl (rebuilt, double atomics) fpnb (fixed point only) fp (fixed point + deterministic blocking)
direct J/K (RYS_build_jk) 7/28 7/28 28/28 28/28
XC potential matrix 15/28 16/28 28/28 28/28
SMD potential (int1e_grids) 7/28 7/28 28/28 28/28
DF 2e gradient (ejk_int3c2e_ip1 + e_int2c2e_ip1) 7/28 7/28 28/28 28/28
SMD charge gradient (int1e_grids_ip1) 7/28 7/28 28/28 28/28
converged SCF energy 12/28 16/28 28/28 28/28
converged density 7/28 7/28 28/28 28/28
total nuclear gradient 7/28 7/28 28/28 28/28
direct per-atom 2e gradient (RYS_per_atom_jk_ip1, 1.7.6 version) 7/28 7/28 25/28 25/28

Master rewrote the last kernel; its master version is bit-identical under
fixed point in our kernel-family test.

Table E2. Wall-time cost of fixed point.
Setup: same job as E1 · fp vs dbl, so only the accumulation differs · median
and range over the 7 molecules · mean of 4 repeats per molecule.

piece median fp/dbl range
SCF 1.014 0.993 to 1.022
gradient 1.004 0.995 to 1.020
DF 2e gradient 0.988 0.984 to 1.007
direct J/K 1.035 0.772 to 1.084
SMD potential (0.1 s absolute) 1.070 0.994 to 1.094

Table E3. Accuracy of fixed point against double.
Setup: same job as E1 · repeat-0 values · max abs difference over the 7
molecules.

quantity max abs(fp - dbl) max abs(dbl - stock)
SCF energy 1.8e-12 Ha 9.1e-13 Ha
direct 2e gradient, identical input density 4.2e-13 Ha/Bohr 5.7e-14 Ha/Bohr
total gradient, each from its own converged SCF 5.6e-8 Ha/Bohr 6.4e-8 Ha/Bohr

The 5e-8 Ha/Bohr total-gradient difference is SCF-state noise carried into
the gradient, not fixed-point rounding: the same kernel on identical inputs
agrees to 4e-13, and fp vs dbl is no larger than dbl vs stock.

Table E4. Full fork patch on upstream master.
Setup: master d0ea5b0 + fork patch · 7 molecules x 3 repeats · production
SCF + forces · login-node A100 · 34 recorded pieces: DF tensor, DF J/K,
direct J/K, direct per-atom gradient, XC, VV10, SMD potential, SCF energy,
density, orbital energies, DF and SMD gradient terms, total forces.

quantity stock master, identical repeats deterministic, identical repeats
SCF energy 10/21 (spread <= 1.8e-12 Ha) 21/21
converged density 7/21 21/21
all 34 pieces incl. total forces not all 21/21 each

Table E5. Analytic Hessians, 10 production molecules.
Setup: master + fork patch · production settings (DF, SMD, level-4 grid) ·
molecules with S/F/N/P, 6 to 15 atoms, 136 to 355 AOs, charges -2 to +2,
singlets and doublets · 2 fresh SCF + Hessian repeats per molecule and mode ·
one A100 per (molecule, mode) · Slurm 60338207 · dH = max |difference|
between the two repeats.

molecule natm nao stock: E identical / H identical / max dH (eV/Ų) det: E identical / H identical / max dH det vs stock, max dH (eV/Ų) max abs(H) (eV/Ų) s per repeat, stock / det
frag_0 9 212 no / no / 9.5e-08 yes / yes / 0 1.0e-07 115.8 22 / 23
frag_51 6 139 no / no / 7.1e-07 yes / yes / 0 9.1e-07 56.6 20 / 21
recomb_746 13 305 no / no / 1.3e-07 yes / yes / 0 1.6e-07 119.4 42 / 42
recomb_2905 6 139 no / no / 5.3e-08 yes / yes / 0 3.2e-07 99.3 22 / 22
recomb_6180 6 139 no / no / 3.9e-08 yes / yes / 0 5.5e-08 73.1 22 / 22
recomb_6573 12 226 no / no / 2.9e-07 yes / yes / 0 3.9e-07 191.1 44 / 44
recomb_13108 15 355 no / no / 3.8e-07 yes / yes / 0 2.2e-07 89.8 55 / 56
recomb_17591 6 136 no / no / 2.9e-08 yes / yes / 0 8.8e-08 80.8 21 / 22
recomb_17788 10 173 no / no / 4.7e-08 yes / yes / 0 6.3e-08 63.3 32 / 33
recomb_18904 13 231 yes / no / 1.9e-07 yes / yes / 0 2.2e-07 147.1 50 / 51
total E 1/10, H 0/10 E 10/10, H 10/10 all < 1e-6

Table E6. Analytic Hessians, three small test molecules.
Setup: master + fork patch · DF, SMD, level-1 grid · 3 fresh SCF + Hessian
repeats per mode · A100 · Slurm 60179426 · spread = max |difference| across
repeats.

molecule stock Hessian spread (Ha/Bohr²) deterministic spread det vs stock (Ha/Bohr²) note
H2O (RKS) 5.0e-11 0 3.5e-11 also bit-identical on a second node
HOFS radical (UKS) 2.4e-10 0 2.5e-10
C2H5OF triplet (UKS) 1.5e-9 0 1.4e-9

Known limitations, for discussion

  • Accumulator range [measured on S and P, assumed beyond]. A 48-bit
    shift holds ±2^15 per accumulator. A single term above that saturates and
    a sum beyond 2^63 wraps, both silently. The nuclear-attraction Hessian
    terms (ejk_int3c2e_ip2 with nuclei as point charges) scale roughly as
    Z·ρ(0): for sulfur the self-block diagonal came out 32998 Ha/Bohr² too
    large; O and F were in range. The fork guards that kernel by scaling the
    point charges down by a power of two chosen from Z_max
    (charge_scale_for_range, 16 for P/S) and the result back up. A headroom
    probe shows S needs >= 2, so the margin is 8x, and Table E5 exercises the
    guard on every S/P case. It is calibrated on S and extrapolated as Z⁴;
    heavier elements (Cl, Br, I, metals) and the other converted Hessian
    kernels are unchecked. Before a PR I would add a device-side overflow flag
    that raises instead of returning a silently wrong number. A smaller shift
    (e.g. 40) would widen the range at the cost of resolution; input on that
    trade-off, or on a per-kernel shift, is welcome.
  • Diff size. About 12k CUDA lines across ~45 .cu files, almost all the
    mechanical atomicAdd → fp_atomic_add substitution in the unrolled
    kernels; ~470 Python lines across 26 files. Alternatives that leave the
    kernels alone (deterministic tree reductions) would need per-kernel
    restructuring and did not look cheaper.
  • Distribution. The _fp libraries roughly double the compiled-library
    size, so a build option may suit better than the default wheel. Either
    way works for us.
  • Scope of validation. One GPU architecture (A100), one CUDA toolkit
    (12.9), molecular DF-DFT only. We expect bit-identity only within one GPU
    model / driver / cuBLAS version, since the contraction libraries pick
    kernels per architecture. [assumed]

Questions

  1. Is an opt-in deterministic mode something you would accept upstream?
  2. If so, as a build-time option (-DBUILD_FIXED_POINT=ON) producing
    parallel _fp libraries, as in the fork, or another mechanism?
  3. Any preference on the range/overflow handling above?
Lingua principale
Cuda
Stelle
371
Fork
108
Merge medio
2g 6h
PR unite (30g)
51

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di pyscf/gpu4pyscf

Tutte le issue di pyscf/gpu4pyscf

Issue simili

Altre issue su Backend & API Design

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.