diagnose flags R-hat only above 1.015 and ESS only below N/1000; documentation says 1.01 and 400

Đang mở
  • Có gì không ổn?
#1,366 0 bình luận 0 reaction 0 người được giao Xem trên GitHub

Chưa có ai nhận issue này.

Đánh giá

Độ khó
4/5
Thời gian dự kiến
3-5 ngày
Mức phù hợp với người mới
48/100
Loại issue
Lỗi
Độ rõ ràng
Khá rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
cpp
Lĩnh vực
cli

Hướng nghiên cứu

Start with src/cmdstan/diagnose.cpp, where this issue identifies the R-hat and ESS thresholds, then compare the behavior with the CmdStan Guide's diagnose page. Decide which thresholds the project intends to support and align the code or documentation; done means the reported checks and recommendations agree. The issue includes reproductions and measured results, but leaves the intended resolution open.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

Summary

The diagnose utility's R-hat and effective-sample-size checks use thresholds that differ from the ones stated in the CmdStan Guide page for diagnose. In CmdStan 2.40.0, diagnose flags a parameter only when its R-hat exceeds 1.01499 (not 1.01), and flags low ESS only when ESS is below 0.001 x the total number of draws (4 for a default 4 x 1000 run), not below 100 x chains (400). As a result, runs that the documentation says should not be used are reported as "Processing complete, no problems detected."

What the documentation says

https://mc-stan.org/docs/cmdstan-guide/diagnose_utility.html (Version 2.40, re-read 2026-11-10):

We recommend running at least four chains by default and only using the sample if R-hat is less than 1.01.

We recommend requiring that the bulk-ESS is greater than 100 times the number of chains. For example, when running four chains, this corresponds to having a rank-normalized effective sample size of at least 400.

The example output on that page also reads "The following parameters had rank-normalized split R-hat greater than 1.01:".

What the code does (CmdStan 2.40.0)

src/cmdstan/diagnose.cpp:

  • line 12: double RHAT_MAX = 1.01499; // round to 1.01
  • line 166: if (n_eff / num_samples < 0.001) where n_eff = min(ess_bulk, ess_tail) and num_samples is draws x chains (line 79)
  • line 171: if (split_rhat > RHAT_MAX) where split_rhat = max(rhat_bulk, rhat_tail)

So R-hat values in (1.01, 1.01499] pass, and any ESS of at least N/1000 passes. The flag message prints RHAT_MAX with default stream precision, so it reads "greater than 1.01499" when it does fire.

Measured consequence

On constructed 4-chain x 1000-draw CSV sets with known R-hat/ESS:

  • 29 of 29 sets with R-hat (as reported by stansummary) in (1.01, 1.0149] were not flagged (100%; Wilson 95% interval 88.3-100%).
  • 36 of 36 sets with minimum ESS below 400 were not flagged (100%; 90.4-100%).
  • The flag boundary sits between R-hat 1.01497 (not flagged) and 1.01500 (flagged).

On 210 real NUTS runs of 8 deliberately difficult posteriors whose correctness was checked against long reference runs (123 runs with wrong posterior estimates, 87 correct), the shipped thresholds passed 3 of the 123 wrong runs (2.4%; 0.8-6.9%), while the documented thresholds (R-hat < 1.01, bulk-ESS > 400) passed 0 of 123 (0-3.0%). Both passed all 87 correct runs. The difference is small but it is entirely in the direction of accepting runs the docs say to reject.

Minimal reproduction

The script below writes four Stan CSV files: 1000 iid N(0,1) draws per chain, with chain c shifted by 0.13 x (c - 1.5).

# make_repro.py  -- python make_repro.py 0.13 8
import sys, numpy as np
d, seed = float(sys.argv[1]), int(sys.argv[2])
rng = np.random.default_rng(seed)
head = "lp__,accept_stat__,stepsize__,treedepth__,n_leapfrog__,divergent__,energy__,theta"
for c in range(4):
    th = rng.normal(size=1000) + d * (c - 1.5)
    en = rng.normal(size=1000) * 2 + 10
    rows = ["# method = sample (Default)", "#     num_samples = 1000", "#     num_warmup = 0",
            head, "# Adaptation terminated", "# Step size = 0.5",
            "# Diagonal elements of inverse mass matrix:", "# 1"]
    rows += [f"{-0.5*t*t:.9g},0.9,0.5,3,7,0,{e:.9g},{t:.9g}" for t, e in zip(th, en)]
    open(f"chain_{c+1}.csv", "w", newline="\n").write("\n".join(rows) + "\n")

$ python make_repro.py 0.13 8
$ bin/stansummary chain_1.csv chain_2.csv chain_3.csv chain_4.csv
           Mean  ...  ESS_bulk  ESS_tail  ...  R_hat
theta  -1.49e-02 ...  2.17e+03  3.39e+03  ...  1.0116
$ bin/diagnose chain_1.csv chain_2.csv chain_3.csv chain_4.csv
...
Rank-normalized split effective sample size satisfactory for all parameters.

Rank-normalized split R-hat values satisfactory for all parameters.

Processing complete, no problems detected.

R-hat 1.0116 is above the documented 1.01, but nothing is flagged. Low ESS: with AR(1) chains (autocorrelation 0.9, no shift, seed 3), stansummary reports ESS_bulk 227 and R_hat 1.0075, and diagnose again prints "no problems detected" (227 > 0.001 x 4000 = 4).

Suggested fix

Either of these would make the tool and the docs agree; we have no preference beyond consistency:

  1. Code to the docs: RHAT_MAX = 1.01 (comparison >= or > as intended), and flag bulk-ESS below 100 * num_chains (possibly tail-ESS too).
  2. Docs to the code: state on the diagnose page that the tool flags R-hat above about 1.015 and ESS below 0.001 per draw, and that the 1.01 / 400 recommendations should be checked separately (e.g. with stansummary).

If 1.01499 is meant to let values that print as "1.01" pass, saying so in the docs would also help.

Environment
  • CmdStan 2.40.0 (conda-forge build), bin/diagnose and bin/stansummary
  • Windows 11, Python 3.14 / numpy to generate the CSVs
  • Source lines cited from the 2.40.0 release tarball
Ngôn ngữ chính
C++
Star
240
Fork
105
Merge trung bình
1 ngày 5 giờ
Pull request đã merge (30 ngày)
1

Chuẩn bị môi trường

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của stan-dev/cmdstan

Tất cả issue của stan-dev/cmdstan

Issue tương tự

Thêm issue về C++

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.