diagnose flags R-hat only above 1.015 and ESS only below N/1000; documentation says 1.01 and 400

Aperta
  • Qualcosa non va?
#1,366 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
48/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
cpp
Ambito
cli

Direzione di ricerca

Start with src/cmdstan/diagnose.cpp, where this issue identifies the R-hat and ESS thresholds, then compare the behavior with the CmdStan Guide's diagnose page. Decide which thresholds the project intends to support and align the code or documentation; done means the reported checks and recommendations agree. The issue includes reproductions and measured results, but leaves the intended resolution open.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Summary

The diagnose utility's R-hat and effective-sample-size checks use thresholds that differ from the ones stated in the CmdStan Guide page for diagnose. In CmdStan 2.40.0, diagnose flags a parameter only when its R-hat exceeds 1.01499 (not 1.01), and flags low ESS only when ESS is below 0.001 x the total number of draws (4 for a default 4 x 1000 run), not below 100 x chains (400). As a result, runs that the documentation says should not be used are reported as "Processing complete, no problems detected."

What the documentation says

https://mc-stan.org/docs/cmdstan-guide/diagnose_utility.html (Version 2.40, re-read 2026-11-10):

We recommend running at least four chains by default and only using the sample if R-hat is less than 1.01.

We recommend requiring that the bulk-ESS is greater than 100 times the number of chains. For example, when running four chains, this corresponds to having a rank-normalized effective sample size of at least 400.

The example output on that page also reads "The following parameters had rank-normalized split R-hat greater than 1.01:".

What the code does (CmdStan 2.40.0)

src/cmdstan/diagnose.cpp:

  • line 12: double RHAT_MAX = 1.01499; // round to 1.01
  • line 166: if (n_eff / num_samples < 0.001) where n_eff = min(ess_bulk, ess_tail) and num_samples is draws x chains (line 79)
  • line 171: if (split_rhat > RHAT_MAX) where split_rhat = max(rhat_bulk, rhat_tail)

So R-hat values in (1.01, 1.01499] pass, and any ESS of at least N/1000 passes. The flag message prints RHAT_MAX with default stream precision, so it reads "greater than 1.01499" when it does fire.

Measured consequence

On constructed 4-chain x 1000-draw CSV sets with known R-hat/ESS:

  • 29 of 29 sets with R-hat (as reported by stansummary) in (1.01, 1.0149] were not flagged (100%; Wilson 95% interval 88.3-100%).
  • 36 of 36 sets with minimum ESS below 400 were not flagged (100%; 90.4-100%).
  • The flag boundary sits between R-hat 1.01497 (not flagged) and 1.01500 (flagged).

On 210 real NUTS runs of 8 deliberately difficult posteriors whose correctness was checked against long reference runs (123 runs with wrong posterior estimates, 87 correct), the shipped thresholds passed 3 of the 123 wrong runs (2.4%; 0.8-6.9%), while the documented thresholds (R-hat < 1.01, bulk-ESS > 400) passed 0 of 123 (0-3.0%). Both passed all 87 correct runs. The difference is small but it is entirely in the direction of accepting runs the docs say to reject.

Minimal reproduction

The script below writes four Stan CSV files: 1000 iid N(0,1) draws per chain, with chain c shifted by 0.13 x (c - 1.5).

# make_repro.py  -- python make_repro.py 0.13 8
import sys, numpy as np
d, seed = float(sys.argv[1]), int(sys.argv[2])
rng = np.random.default_rng(seed)
head = "lp__,accept_stat__,stepsize__,treedepth__,n_leapfrog__,divergent__,energy__,theta"
for c in range(4):
    th = rng.normal(size=1000) + d * (c - 1.5)
    en = rng.normal(size=1000) * 2 + 10
    rows = ["# method = sample (Default)", "#     num_samples = 1000", "#     num_warmup = 0",
            head, "# Adaptation terminated", "# Step size = 0.5",
            "# Diagonal elements of inverse mass matrix:", "# 1"]
    rows += [f"{-0.5*t*t:.9g},0.9,0.5,3,7,0,{e:.9g},{t:.9g}" for t, e in zip(th, en)]
    open(f"chain_{c+1}.csv", "w", newline="\n").write("\n".join(rows) + "\n")

$ python make_repro.py 0.13 8
$ bin/stansummary chain_1.csv chain_2.csv chain_3.csv chain_4.csv
           Mean  ...  ESS_bulk  ESS_tail  ...  R_hat
theta  -1.49e-02 ...  2.17e+03  3.39e+03  ...  1.0116
$ bin/diagnose chain_1.csv chain_2.csv chain_3.csv chain_4.csv
...
Rank-normalized split effective sample size satisfactory for all parameters.

Rank-normalized split R-hat values satisfactory for all parameters.

Processing complete, no problems detected.

R-hat 1.0116 is above the documented 1.01, but nothing is flagged. Low ESS: with AR(1) chains (autocorrelation 0.9, no shift, seed 3), stansummary reports ESS_bulk 227 and R_hat 1.0075, and diagnose again prints "no problems detected" (227 > 0.001 x 4000 = 4).

Suggested fix

Either of these would make the tool and the docs agree; we have no preference beyond consistency:

  1. Code to the docs: RHAT_MAX = 1.01 (comparison >= or > as intended), and flag bulk-ESS below 100 * num_chains (possibly tail-ESS too).
  2. Docs to the code: state on the diagnose page that the tool flags R-hat above about 1.015 and ESS below 0.001 per draw, and that the 1.01 / 400 recommendations should be checked separately (e.g. with stansummary).

If 1.01499 is meant to let values that print as "1.01" pass, saying so in the docs would also help.

Environment
  • CmdStan 2.40.0 (conda-forge build), bin/diagnose and bin/stansummary
  • Windows 11, Python 3.14 / numpy to generate the CSVs
  • Source lines cited from the 2.40.0 release tarball
Lingua principale
C++
Stelle
240
Fork
105
Merge medio
1g 5h
PR unite (30g)
1

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di stan-dev/cmdstan

Tutte le issue di stan-dev/cmdstan

Issue simili

Altre issue su C++

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.