Run-to-run non-determinism from floating-point atomics
メンテナーはふだん 2 日以内に返信
まだ誰も着手していません。
評価
- 難易度
- 5/5
- 見積もり時間
- 1週間以上
- 初心者へのやさしさ
- 35/100
調査の方向性
Start with the fork’s DETERMINISM.md and gpu4pyscf/lib/tests/test_fixed_point.py to understand the proposed approach and existing checks. Review the listed kernel families and the unvalidated PBC, TDDFT, and direct-SCF Hessian paths to establish what an upstream change would need to cover. Done should include an agreed scope and reproducible tests showing the intended paths are bit-identical across runs.
索引モデルが issue の本文から書いたものです。
説明
Summary
Repeated runs of the same input on the same GPU give slightly different
energies, gradients and analytic Hessians. The dominant cause is the
double-precision atomicAdd reductions in the CUDA kernels, whose summation
order depends on thread scheduling. The spread is far below the DF, grid and
SCF convergence error, so this is a reproducibility problem, not an accuracy
one. It still rules out bit-for-bit regression testing and bisecting. Our
case: geometry optimizations of open- and closed-shell molecules (Sella,
gpu4pyscf forces). The optimizer amplifies the 2e-7 eV/Å force noise by
about 1.9x per step, so identical repeat optimizations ended in different
minima 23 % of the time (up to 23 kcal/mol apart), and we could not separate
optimizer bugs from floating-point noise until the forces were reproducible.
I have an opt-in fix working in a fork and would like to know whether a PR is
welcome, and in what form, before cleaning it up.
Claims are marked [measured] when a run backs them (see Evidence) and
[assumed] when inferred from code or documentation but not isolated in a
test.
Observed behaviour [measured]
Table 1. Run-to-run differences in stock gpu4pyscf.
Setup: DF wB97M-V/def2-SVPD + SMD · A100-SXM4 40 GB and 80 GB · CUDA 12.9 ·
gpu4pyscf 1.7.6, 1.8.1 and master (d0ea5b0) · 6 to 19 atom molecules with
H, C, N, O, F, P, S · fresh process per repeat.
| Quantity | Run-to-run difference | Sample |
|---|---|---|
| SCF total energy | median 3e-13 Ha, 22 % of pairs bit-identical, max 2e-12 Ha | 192 repeat pairs from a production campaign (1.7.6); 48 same-GPU repeats; 20 repeats on master |
| Nuclear gradient | RMS 2e-7 eV/Å (p90 9e-7), max component ~1e-6 eV/Å, 0 of 240 pairs identical | same |
| Analytic Hessian | 3e-8 to 7e-7 eV/Ų (5e-11 to 1.5e-9 Ha/Bohr²), 0 of 13 molecules identical | 10 molecules x 2 repeats + 3 molecules x 3 repeats, master |
Same-process repeats on one GPU and repeats across the four GPUs of a node
give the same spread (|dE| <= 9e-13 Ha, force RMS 1 to 2e-7 eV/Å), so this is
not a device-to-device effect. The energy is occasionally identical because
the XC quadrature runs through cuBLAS/cuTENSOR, which are deterministic; the
J/K, DF-gradient and solvent kernels, which use atomics, agreed in 0 of 21 to
28 repeats in every benchmark.
CPU PySCF has the same class of problem [measured] (Table 2, last row).
Single-threaded runs, or OpenMP at 1 thread with OpenBLAS at 8, are
bit-identical; OpenMP at 8 with OpenBLAS at 1 is not. The source is
PySCF's own OpenMP code: per-thread partial sums merged under
#pragma omp critical in arrival order, e.g. VXCdot_ao_ao in
lib/dft/nr_numint.c (XC matrix, 1e-15 on a fixed density) and the K-split
NPdgemm behind lib.dot in lib/np_helper/npdot.c (DF J/K, 1e-15). On
CPU an ordered merge of the thread buffers would fix it, and PySCF already
has one (NPomp_dsum_reduce_inplace). This issue is about the GPU path,
where thousands of concurrent adders make ordered merging impractical and
fixed point is the natural tool.
Not measured: translational/rotational invariance of the gradient beyond
this noise level.
Reproducer
import sys, numpy as np, pyscf
from gpu4pyscf import dft
mol = pyscf.M(atom="O 0 0 0; H 0 0.76 0.59; H 0 -0.76 0.59; F 1.8 0 0.2; S 1.2 1.5 -0.4",
basis="def2-svp", spin=1, verbose=0)
mf = dft.UKS(mol, xc="wb97x").density_fit()
mf.grids.level = 1
mf.conv_tol = 1e-10
mf.kernel()
g = mf.nuc_grad_method().kernel()
h = mf.Hessian().kernel()
np.savez(sys.argv[1], e=mf.e_tot, g=g, h=h)
Run as separate processes and compare the .npz files.
Table 2. Reproducer results.
Setup: 3 runs per row, separate processes · shared login-node
A100-PCIE-40GB · driver 580.178.04 · CUDA runtime 12.9 · PySCF 2.14.0 ·
CuPy 14.1.1 · CPU row: 8 OpenMP threads · values are max |difference|
between runs · largest gradient component 0.21 Ha/Bohr, largest Hessian
element 1.03 Ha/Bohr².
| build | max ΔE (Ha) | max Δgradient (Ha/Bohr) | max ΔHessian (Ha/Bohr²) |
|---|---|---|---|
| gpu4pyscf-cuda12x 1.7.6 wheel | 0 (3/3 identical) | 7e-15 to 1.8e-14 | 0 (3/3 identical) |
| fork, upstream 1.8 base (d0ea5b0), deterministic mode off | 1.1e-13 | 4.9e-10 to 9.8e-10 | 1.4e-10 to 1.5e-10 |
fork, GPU4PYSCF_DETERMINISTIC=1 |
0 (3/3 identical) | 0 (3/3 identical) | 0 (3/3 identical) |
| fork, deterministic vs non-deterministic (same build) | 1.1e-12 | 5.0e-10 | 2.3e-10 |
| PySCF 2.14.0 on CPU, 8 threads | 1.1e-13 to 6.8e-13 | 1.9e-8 to 2.3e-8 | 3.5e-9 to 3.8e-9 |
Two cautions. The reproducer is small and gas-phase; on the 1.7.6 wheel only
the gradient moved. The spread grows with system size and with the number of
atomic kernels on the path (SMD, DF gradient), hence the larger numbers in
Table 1. And three runs is a small sample: in production the J/K,
DF-gradient and solvent kernels never agreed in any of 28 repeats, so two
identical runs do not disprove the effect.
Where the non-determinism comes from
Demonstrated sources [measured]
atomicAdd(double*, double)in the integral kernels. Controlled A/B:
same source, same compiler, double atomics (dbl) vs int64 fixed-point
atomics (fp), nothing else changed. Kernels with plain-array inputs go
from "no two repeats agree" to "all agree" (Table E1). Libraries and
kernels, from the fork's diff:libgvhf_rys: J/K contraction and theip1/ip2derivative kernels
(rys_contract_jk*.cu,unrolled_rys_jk*.cu,unrolled_ejk_ip*.cu,
ejk_int3c2e_ip*.cu,e_int2c2e_ip2.cu); validatedlibgvhf_md: J engine (md_contract_j.cu,unrolled_md_j*.cu);
validated via the DF J path on masterlibgint:int3c1e*andint1e_grids; validated via the SMD
potential and its charge gradientlibpbc:rys_contract_j/k,unrolled_rys_k.cu,e_int2c2e_ip1.cu;
converted because they sharedot_dmwith the Rys kernels, NOT
validated (we run no PBC)
- Shared-memory race in
_screen_index(lib/gdft/nr_eval_gto.cu).
A freshly built Grids object keeps or drops a borderline shell at random,
so the XC matrix differs at cutoff level. Upstream issue #919, fixed in
PR #925 after 1.8.1. Confirmed by comparing the screening mask of
repeatedly built Grids objects: differs on 1.8.1, identical 21/21 on
master. - Duplicate AO pairs in the packed DF tensor. On master the packed
cderistores some AO pairs as bothijandji(224 of 18,290 for a
12-atom molecule) with values differing at 1e-13, and
decompress_and_filllets the last block win. With every kernel already
integer-accumulated this alone left the SCF energy varying at <= 1.8e-12
Ha (10/21 identical); a first-occurrence write mask gave 21/21.
Likely sources, not isolated [assumed]
cupy.add.atscatters ingpu4pyscf/hessian/{rhf,rks,uks}.py. CuPy
implements these with float atomics. We replaced all 14 calls with a
fixed-order one-hot matrix product and the Hessian became bit-identical,
but we never tested the Hessian with onlycupy.add.atleft in place, so
its own contribution is unknown.- Block and batch sizes derived from current free memory
(cupy_helperavail-mem queries), which make the reduction grouping
depend on the memory-pool state. Replaced by a fixed fraction of total
memory. Before the screening race was fixed this moved the SCF energy from
18/28 to 22/28 identical and the DF gradient from 17/28 to 19/28; after
the fix both settings reach 28/28, so the effect was never isolated.
Proposed fix (implemented in fork)
Fork: https://github.com/oliverhvidsten/gpu4pyscf (branch master, tag
v1.8.1-det.1, on upstream d0ea5b0). Write-up:
https://github.com/oliverhvidsten/gpu4pyscf/blob/master/DETERMINISM.md.
Tests: gpu4pyscf/lib/tests/test_fixed_point.py (7 tests, including a
two-process bit-identity check).
Opt-in via GPU4PYSCF_DETERMINISTIC=1; default behaviour unchanged.
- Fixed-point accumulation (AMBER SPFP scheme).
lib/fixed_point.cuh
definesfp_atomic_add(double*, double). With-DFIXED_POINT_ACCUMit
scales by 2^48, rounds toint64and does an integeratomicAddinto the
same 8-byte slot; integer addition is associative, so the result is
order-independent. Without the define it is the plain doubleatomicAdd.
The four libraries are built a second time aslib*_fp.so, and the
Python call sites convert the buffer back once withfrom_fixed(). - Deterministic blocking. Block sizes from a fixed fraction of total
memory; thecderiunpack writes each AO pair once;cupy.add.at
replaced by a one-hot matrix product (lib/fixed_point.add_at) with a
fixed summation order.
Results [measured, Tables E1 to E5]:
- Bit-identical across repeats: SCF energy, density, orbital energies,
forces and analytic Hessian, every repeat of every molecule (28/28, 21/21
and 10/10 in three independent runs). - Across nodes of the same GPU model: one data point (H2O Hessian, two A100
nodes). [assumed beyond that] - Agreement with the stock build: 1e-12 Ha in the SCF energy, 4e-13 Ha/Bohr
for a single gradient kernel on identical inputs, 3e-11 to 1.4e-9 Ha/Bohr²
in the Hessian; all within the stock build's own spread. - Wall time, deterministic/stock: 0.99 to 1.02 for SCF and gradient, 1.00 to
1.05 for SCF + Hessian; one 0.1 s piece (SMD potential) at 1.07.
Validated: molecular DF-DFT (RKS/UKS) with SMD, SCF, gradient and analytic
Hessian, on H, C, N, O, F, P, S. PBC, TDDFT and direct-SCF Hessian paths have
the fixed-point kernels but are unvalidated.
Evidence
All runs on Perlmutter A100 nodes with fresh SCFs. "n/28" = n of 28
(7 molecules x 4 repeats) repeat values bit-identical to the first repeat of
their molecule; 7/28 means no two repeats ever agreed.
Table E1. Controlled A/B: bit-identical repeats per piece.
Setup: same source (1.7.6 kernels + the #925 screening fix) and same
compiler for dbl, fpnb, fp · A100-SXM4-80GB · 7 molecules x 4 repeats ·
one process per variant · Slurm 59791528.
| piece | stock (released wheel) | dbl (rebuilt, double atomics) | fpnb (fixed point only) | fp (fixed point + deterministic blocking) |
|---|---|---|---|---|
direct J/K (RYS_build_jk) |
7/28 | 7/28 | 28/28 | 28/28 |
| XC potential matrix | 15/28 | 16/28 | 28/28 | 28/28 |
SMD potential (int1e_grids) |
7/28 | 7/28 | 28/28 | 28/28 |
DF 2e gradient (ejk_int3c2e_ip1 + e_int2c2e_ip1) |
7/28 | 7/28 | 28/28 | 28/28 |
SMD charge gradient (int1e_grids_ip1) |
7/28 | 7/28 | 28/28 | 28/28 |
| converged SCF energy | 12/28 | 16/28 | 28/28 | 28/28 |
| converged density | 7/28 | 7/28 | 28/28 | 28/28 |
| total nuclear gradient | 7/28 | 7/28 | 28/28 | 28/28 |
direct per-atom 2e gradient (RYS_per_atom_jk_ip1, 1.7.6 version) |
7/28 | 7/28 | 25/28 | 25/28 |
Master rewrote the last kernel; its master version is bit-identical under
fixed point in our kernel-family test.
Table E2. Wall-time cost of fixed point.
Setup: same job as E1 · fp vs dbl, so only the accumulation differs · median
and range over the 7 molecules · mean of 4 repeats per molecule.
| piece | median fp/dbl | range |
|---|---|---|
| SCF | 1.014 | 0.993 to 1.022 |
| gradient | 1.004 | 0.995 to 1.020 |
| DF 2e gradient | 0.988 | 0.984 to 1.007 |
| direct J/K | 1.035 | 0.772 to 1.084 |
| SMD potential (0.1 s absolute) | 1.070 | 0.994 to 1.094 |
Table E3. Accuracy of fixed point against double.
Setup: same job as E1 · repeat-0 values · max abs difference over the 7
molecules.
| quantity | max abs(fp - dbl) | max abs(dbl - stock) |
|---|---|---|
| SCF energy | 1.8e-12 Ha | 9.1e-13 Ha |
| direct 2e gradient, identical input density | 4.2e-13 Ha/Bohr | 5.7e-14 Ha/Bohr |
| total gradient, each from its own converged SCF | 5.6e-8 Ha/Bohr | 6.4e-8 Ha/Bohr |
The 5e-8 Ha/Bohr total-gradient difference is SCF-state noise carried into
the gradient, not fixed-point rounding: the same kernel on identical inputs
agrees to 4e-13, and fp vs dbl is no larger than dbl vs stock.
Table E4. Full fork patch on upstream master.
Setup: master d0ea5b0 + fork patch · 7 molecules x 3 repeats · production
SCF + forces · login-node A100 · 34 recorded pieces: DF tensor, DF J/K,
direct J/K, direct per-atom gradient, XC, VV10, SMD potential, SCF energy,
density, orbital energies, DF and SMD gradient terms, total forces.
| quantity | stock master, identical repeats | deterministic, identical repeats |
|---|---|---|
| SCF energy | 10/21 (spread <= 1.8e-12 Ha) | 21/21 |
| converged density | 7/21 | 21/21 |
| all 34 pieces incl. total forces | not all | 21/21 each |
Table E5. Analytic Hessians, 10 production molecules.
Setup: master + fork patch · production settings (DF, SMD, level-4 grid) ·
molecules with S/F/N/P, 6 to 15 atoms, 136 to 355 AOs, charges -2 to +2,
singlets and doublets · 2 fresh SCF + Hessian repeats per molecule and mode ·
one A100 per (molecule, mode) · Slurm 60338207 · dH = max |difference|
between the two repeats.
| molecule | natm | nao | stock: E identical / H identical / max dH (eV/Ų) | det: E identical / H identical / max dH | det vs stock, max dH (eV/Ų) | max abs(H) (eV/Ų) | s per repeat, stock / det |
|---|---|---|---|---|---|---|---|
| frag_0 | 9 | 212 | no / no / 9.5e-08 | yes / yes / 0 | 1.0e-07 | 115.8 | 22 / 23 |
| frag_51 | 6 | 139 | no / no / 7.1e-07 | yes / yes / 0 | 9.1e-07 | 56.6 | 20 / 21 |
| recomb_746 | 13 | 305 | no / no / 1.3e-07 | yes / yes / 0 | 1.6e-07 | 119.4 | 42 / 42 |
| recomb_2905 | 6 | 139 | no / no / 5.3e-08 | yes / yes / 0 | 3.2e-07 | 99.3 | 22 / 22 |
| recomb_6180 | 6 | 139 | no / no / 3.9e-08 | yes / yes / 0 | 5.5e-08 | 73.1 | 22 / 22 |
| recomb_6573 | 12 | 226 | no / no / 2.9e-07 | yes / yes / 0 | 3.9e-07 | 191.1 | 44 / 44 |
| recomb_13108 | 15 | 355 | no / no / 3.8e-07 | yes / yes / 0 | 2.2e-07 | 89.8 | 55 / 56 |
| recomb_17591 | 6 | 136 | no / no / 2.9e-08 | yes / yes / 0 | 8.8e-08 | 80.8 | 21 / 22 |
| recomb_17788 | 10 | 173 | no / no / 4.7e-08 | yes / yes / 0 | 6.3e-08 | 63.3 | 32 / 33 |
| recomb_18904 | 13 | 231 | yes / no / 1.9e-07 | yes / yes / 0 | 2.2e-07 | 147.1 | 50 / 51 |
| total | E 1/10, H 0/10 | E 10/10, H 10/10 | all < 1e-6 |
Table E6. Analytic Hessians, three small test molecules.
Setup: master + fork patch · DF, SMD, level-1 grid · 3 fresh SCF + Hessian
repeats per mode · A100 · Slurm 60179426 · spread = max |difference| across
repeats.
| molecule | stock Hessian spread (Ha/Bohr²) | deterministic spread | det vs stock (Ha/Bohr²) | note |
|---|---|---|---|---|
| H2O (RKS) | 5.0e-11 | 0 | 3.5e-11 | also bit-identical on a second node |
| HOFS radical (UKS) | 2.4e-10 | 0 | 2.5e-10 | |
| C2H5OF triplet (UKS) | 1.5e-9 | 0 | 1.4e-9 |
Known limitations, for discussion
- Accumulator range [measured on S and P, assumed beyond]. A 48-bit
shift holds ±2^15 per accumulator. A single term above that saturates and
a sum beyond 2^63 wraps, both silently. The nuclear-attraction Hessian
terms (ejk_int3c2e_ip2with nuclei as point charges) scale roughly as
Z·ρ(0): for sulfur the self-block diagonal came out 32998 Ha/Bohr² too
large; O and F were in range. The fork guards that kernel by scaling the
point charges down by a power of two chosen from Z_max
(charge_scale_for_range, 16 for P/S) and the result back up. A headroom
probe shows S needs >= 2, so the margin is 8x, and Table E5 exercises the
guard on every S/P case. It is calibrated on S and extrapolated as Z⁴;
heavier elements (Cl, Br, I, metals) and the other converted Hessian
kernels are unchecked. Before a PR I would add a device-side overflow flag
that raises instead of returning a silently wrong number. A smaller shift
(e.g. 40) would widen the range at the cost of resolution; input on that
trade-off, or on a per-kernel shift, is welcome. - Diff size. About 12k CUDA lines across ~45
.cufiles, almost all the
mechanicalatomicAdd→fp_atomic_addsubstitution in the unrolled
kernels; ~470 Python lines across 26 files. Alternatives that leave the
kernels alone (deterministic tree reductions) would need per-kernel
restructuring and did not look cheaper. - Distribution. The
_fplibraries roughly double the compiled-library
size, so a build option may suit better than the default wheel. Either
way works for us. - Scope of validation. One GPU architecture (A100), one CUDA toolkit
(12.9), molecular DF-DFT only. We expect bit-identity only within one GPU
model / driver / cuBLAS version, since the contraction libraries pick
kernels per architecture. [assumed]
Questions
- Is an opt-in deterministic mode something you would accept upstream?
- If so, as a build-time option (
-DBUILD_FIXED_POINT=ON) producing
parallel_fplibraries, as in the fork, or another mechanism? - Any preference on the range/overflow handling above?
- 主要言語
- Cuda
- スター
- 371
- フォーク
- 108
- 平均マージ
- 2日 6時間
- マージ済み PR(30日)
- 51
環境構築
- Dockerfile・Docker Compose ファイルなし
- プルリクエストのテンプレートなし
- コントリビューションガイドを読む
はじめの一歩
- issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
- 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
- リポジトリをフォークし、ブランチを切って変更します。
- issue 番号を参照したプルリクエストを送ります。
pyscf/gpu4pyscf のほかの issue
-
難易度 3/5 1〜2日 初心者へのやさしさ 30/100
メンテナーはふだん 2 日以内に返信
-
難易度 3/5 半日 初心者へのやさしさ 35/100
メンテナーはふだん 2 日以内に返信
-
難易度 4/5 3〜5日 初心者へのやさしさ 35/100
メンテナーはふだん 2 日以内に返信
-
難易度 4/5 3〜5日 初心者へのやさしさ 55/100
メンテナーはふだん 2 日以内に返信
-
難易度 3/5 1〜2日 初心者へのやさしさ 68/100
メンテナーはふだん 2 日以内に返信
pyscf/gpu4pyscf の issue をすべて見る
似ている issue
-
難易度 2/5 1〜3時間 初心者へのやさしさ 72/100
umbraco/Umbraco.Automate#612 ·
メンテナーはふだん 1 日以内に返信
-
Setup reports an unknown outcome after saving the space when its log cannot open対応中かも このイシューにリンクされたプルリクエストがオープン中、またはマージ済みです。 オープンbug triage:owned
難易度 2/5 1〜3時間 初心者へのやさしさ 72/100
メンテナーはふだん 1 日以内に返信
-
backlog-skill bot cat:tech-debt effort:s ready-for-agent refactor sev:low
難易度 2/5 1〜3時間 初心者へのやさしさ 72/100
LucasSantana-Dev/Lucky#3283 ·
メンテナーはふだん 1 日以内に返信
-
難易度 2/5 半日 初心者へのやさしさ 74/100
メンテナーはふだん 1 日以内に返信
-
tech-debt
難易度 2/5 1〜3時間 初心者へのやさしさ 62/100
メンテナーはふだん 1 日以内に返信