diagnose flags R-hat only above 1.015 and ESS only below N/1000; documentation says 1.01 and 400

Abierto
  • ¿Algo no está bien?
#1,366 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
48/100
Tipo de issue
Error
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
cpp
Área
cli

Línea de trabajo

Start with src/cmdstan/diagnose.cpp, where this issue identifies the R-hat and ESS thresholds, then compare the behavior with the CmdStan Guide's diagnose page. Decide which thresholds the project intends to support and align the code or documentation; done means the reported checks and recommendations agree. The issue includes reproductions and measured results, but leaves the intended resolution open.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Summary

The diagnose utility's R-hat and effective-sample-size checks use thresholds that differ from the ones stated in the CmdStan Guide page for diagnose. In CmdStan 2.40.0, diagnose flags a parameter only when its R-hat exceeds 1.01499 (not 1.01), and flags low ESS only when ESS is below 0.001 x the total number of draws (4 for a default 4 x 1000 run), not below 100 x chains (400). As a result, runs that the documentation says should not be used are reported as "Processing complete, no problems detected."

What the documentation says

https://mc-stan.org/docs/cmdstan-guide/diagnose_utility.html (Version 2.40, re-read 2026-11-10):

We recommend running at least four chains by default and only using the sample if R-hat is less than 1.01.

We recommend requiring that the bulk-ESS is greater than 100 times the number of chains. For example, when running four chains, this corresponds to having a rank-normalized effective sample size of at least 400.

The example output on that page also reads "The following parameters had rank-normalized split R-hat greater than 1.01:".

What the code does (CmdStan 2.40.0)

src/cmdstan/diagnose.cpp:

  • line 12: double RHAT_MAX = 1.01499; // round to 1.01
  • line 166: if (n_eff / num_samples < 0.001) where n_eff = min(ess_bulk, ess_tail) and num_samples is draws x chains (line 79)
  • line 171: if (split_rhat > RHAT_MAX) where split_rhat = max(rhat_bulk, rhat_tail)

So R-hat values in (1.01, 1.01499] pass, and any ESS of at least N/1000 passes. The flag message prints RHAT_MAX with default stream precision, so it reads "greater than 1.01499" when it does fire.

Measured consequence

On constructed 4-chain x 1000-draw CSV sets with known R-hat/ESS:

  • 29 of 29 sets with R-hat (as reported by stansummary) in (1.01, 1.0149] were not flagged (100%; Wilson 95% interval 88.3-100%).
  • 36 of 36 sets with minimum ESS below 400 were not flagged (100%; 90.4-100%).
  • The flag boundary sits between R-hat 1.01497 (not flagged) and 1.01500 (flagged).

On 210 real NUTS runs of 8 deliberately difficult posteriors whose correctness was checked against long reference runs (123 runs with wrong posterior estimates, 87 correct), the shipped thresholds passed 3 of the 123 wrong runs (2.4%; 0.8-6.9%), while the documented thresholds (R-hat < 1.01, bulk-ESS > 400) passed 0 of 123 (0-3.0%). Both passed all 87 correct runs. The difference is small but it is entirely in the direction of accepting runs the docs say to reject.

Minimal reproduction

The script below writes four Stan CSV files: 1000 iid N(0,1) draws per chain, with chain c shifted by 0.13 x (c - 1.5).

# make_repro.py  -- python make_repro.py 0.13 8
import sys, numpy as np
d, seed = float(sys.argv[1]), int(sys.argv[2])
rng = np.random.default_rng(seed)
head = "lp__,accept_stat__,stepsize__,treedepth__,n_leapfrog__,divergent__,energy__,theta"
for c in range(4):
    th = rng.normal(size=1000) + d * (c - 1.5)
    en = rng.normal(size=1000) * 2 + 10
    rows = ["# method = sample (Default)", "#     num_samples = 1000", "#     num_warmup = 0",
            head, "# Adaptation terminated", "# Step size = 0.5",
            "# Diagonal elements of inverse mass matrix:", "# 1"]
    rows += [f"{-0.5*t*t:.9g},0.9,0.5,3,7,0,{e:.9g},{t:.9g}" for t, e in zip(th, en)]
    open(f"chain_{c+1}.csv", "w", newline="\n").write("\n".join(rows) + "\n")

$ python make_repro.py 0.13 8
$ bin/stansummary chain_1.csv chain_2.csv chain_3.csv chain_4.csv
           Mean  ...  ESS_bulk  ESS_tail  ...  R_hat
theta  -1.49e-02 ...  2.17e+03  3.39e+03  ...  1.0116
$ bin/diagnose chain_1.csv chain_2.csv chain_3.csv chain_4.csv
...
Rank-normalized split effective sample size satisfactory for all parameters.

Rank-normalized split R-hat values satisfactory for all parameters.

Processing complete, no problems detected.

R-hat 1.0116 is above the documented 1.01, but nothing is flagged. Low ESS: with AR(1) chains (autocorrelation 0.9, no shift, seed 3), stansummary reports ESS_bulk 227 and R_hat 1.0075, and diagnose again prints "no problems detected" (227 > 0.001 x 4000 = 4).

Suggested fix

Either of these would make the tool and the docs agree; we have no preference beyond consistency:

  1. Code to the docs: RHAT_MAX = 1.01 (comparison >= or > as intended), and flag bulk-ESS below 100 * num_chains (possibly tail-ESS too).
  2. Docs to the code: state on the diagnose page that the tool flags R-hat above about 1.015 and ESS below 0.001 per draw, and that the 1.01 / 400 recommendations should be checked separately (e.g. with stansummary).

If 1.01499 is meant to let values that print as "1.01" pass, saying so in the docs would also help.

Environment
  • CmdStan 2.40.0 (conda-forge build), bin/diagnose and bin/stansummary
  • Windows 11, Python 3.14 / numpy to generate the CSVs
  • Source lines cited from the 2.40.0 release tarball
Lenguaje dominante
C++
Estrellas
240
Forks
105
Merge medio
1 d 5 h
PR fusionados (30 d)
1

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de stan-dev/cmdstan

Todos los issues de stan-dev/cmdstan

Issues similares

Más issues de C++

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.