[Bug Report] `success_rate_ci` returns wrong Clopper-Pearson bounds at high success rates or large n (binomial CDF underflows)

Đã đóng
  • Có gì không ổn?
#1,926 2 bình luận 0 reaction 1 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

@bercedev đang làm issue này rồi.

Từ ngày 21/10/2026.

  • #1931 của @bercedev — đã merge

Đánh giá

Issue này chưa được đánh giá.

Mô tả

bug complexity-moderate good first issue help wanted TransformerBridge

Describe the bug

transformer_lens.utilities.success_rate_ci silently returns incorrect "exact" Clopper-Pearson intervals once (1 - p) ** n underflows to 0.0 inside _binomial_cdf (transformer_lens/utilities/statistics.py). This happens with as few as 200 trials when the success rate is high, and at around n ≈ 980 for a 50% rate. The resulting intervals can be far too wide, or can even exclude the observed success rate.

Root cause: _binomial_cdf accumulates the pmf by recurrence starting from term = (1.0 - p) ** n. When n * log(1 - p) < ~-745, that starting term is 0.0 in float64, so every later term is also 0.0 and the function returns 0.0 instead of the true CDF. For example, _binomial_cdf(1000, 2000, 0.5) returns 0.0, but the true value is ≈ 0.509. The docstring says the recurrence avoids overflow at large n, but the starting term underflows instead. The bisection in _clopper_pearson_bound then follows the wrong CDF, which corrupts both the lower bound (via 1 - cdf(k - 1)) and the upper bound.

Code example

from transformer_lens.utilities import success_rate_ci
from transformer_lens.utilities.statistics import _binomial_cdf

print(_binomial_cdf(1000, 2000, 0.5))  # 0.0  (true value ~0.509)

for k, n in [(150, 150), (200, 200), (199, 200), (297, 300), (1000, 1000), (1000, 2000), (1990, 2000)]:
    r = success_rate_ci([True] * k + [False] * (n - k))
    print(f"{k}/{n}: rate={r.point_estimate:.4f}  CI=[{r.ci_low:.4f}, {r.ci_high:.4f}]")

Output on current dev (4a92ca50), compared with the exact Clopper-Pearson bounds from scipy.stats.beta.ppf:

successes / trials observed rate success_rate_ci (95%) correct (95%)
150 / 150 1.0000 [0.9757, 1.0000] [0.9757, 1.0000] ✅
200 / 200 1.0000 [0.9759, 1.0000] [0.9817, 1.0000] ❌
199 / 200 0.9950 [0.9725, 0.9759] [0.9725, 0.9999] ❌ (excludes the observed rate)
297 / 300 0.9900 [0.9166, 0.9166] [0.9711, 0.9979] ❌ (excludes the observed rate)
1000 / 1000 1.0000 [0.5243, 1.0000] [0.9963, 1.0000] ❌
1000 / 2000 0.5000 [0.3103, 0.3110] [0.4779, 0.5221] ❌ (excludes the observed rate)
1990 / 2000 0.9950 [0.3103, 0.3110] [0.9908, 0.9976] ❌ (excludes the observed rate)

Reference values:

from scipy.stats import beta
lo = beta.ppf(0.025, k, n - k + 1)   # 0.0 when k == 0
hi = beta.ppf(0.975, k + 1, n - k)   # 1.0 when k == n

At a 50% rate, the first wrong result appears at n = 978: the upper bound is 0.5332 where the correct value is 0.5318.

Expected behavior

Exact Clopper-Pearson bounds for any n, matching scipy.stats.beta.ppf. At minimum, the interval should always contain the point estimate.

Possible fix

Compute the CDF in log space, so no individual term underflows before it is summed. For example, sum exp(log_pmf_i - max_log_pmf) with log_pmf_i = lgamma(n+1) - lgamma(i+1) - lgamma(n-i+1) + i*log(p) + (n-i)*log1p(-p), then rescale. I checked this locally by patching _binomial_cdf with such a log-space version:

  • the CDF matched scipy.stats.binom.cdf to within ~1e-12 across n ∈ {10, 200, 2000} and p ∈ {0.01, 0.5, 0.97, 0.999};
  • every interval in the table above matched the scipy reference to within ~2e-13;
  • the docstring example (0/6 gives [0.0, 0.459]) was unchanged.

A unit test comparing success_rate_ci against a few precomputed Clopper-Pearson values at n = 200–2000 would guard against this regressing.

System Info

  • transformer_lens installed from source (uv sync), dev branch at 4a92ca50
  • OS: Linux (x86_64)
  • Python 3.12.14, torch 2.11.0

Additional context

success_rate_ci is public API (transformer_lens.utilities.success_rate_ci), and it is used to report top-1 success rates in the Jacobian lens causal-swap benchmark. Benchmarks with a few hundred trials and a high success rate are exactly the regime where the interval goes wrong. The function's docstring recommends it over the bootstrap precisely because it is exact, so users are unlikely to suspect its output.

Checklist
  • I have checked that there is no similar issue in the repo (required)
Ngôn ngữ chính
Python
Star
3.9k
Fork
708
Merge trung bình
1 ngày 12 giờ
Pull request đã merge (30 ngày)
79

Chuẩn bị môi trường

Mở trong Codespaces

Khởi chạy dev container của dự án ngay trên trình duyệt, bằng tài khoản GitHub của bạn.

  • Không có Dockerfile hay tệp Docker Compose
  • Có mẫu pull request
  • Không có hướng dẫn đóng góp

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của TransformerLensOrg/TransformerLens

Tất cả issue của TransformerLensOrg/TransformerLens

Issue tương tự

Thêm issue về Python

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.