Hacktoberfest 2026：メンテナが10月に向けて印を付けた、オープンで初心者向けの issue。 Hacktoberfest の issue を見る

[Bug Report] `success_rate_ci` returns wrong Clopper-Pearson bounds at high success rates or large n (binomial CDF underflows)

クローズ
  • 何か問題がありますか？
#1,926 コメント 2 件 リアクション 0 件 担当者 1 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

@bercedev がすでに取り組んでいます。

2026年10月21日 から。

  • #1931 @bercedev による — マージ済み

評価

この issue はまだ評価されていません。

説明

bug complexity-moderate good first issue help wanted TransformerBridge

Describe the bug

transformer_lens.utilities.success_rate_ci silently returns incorrect "exact" Clopper-Pearson intervals once (1 - p) ** n underflows to 0.0 inside _binomial_cdf (transformer_lens/utilities/statistics.py). This happens with as few as 200 trials when the success rate is high, and at around n ≈ 980 for a 50% rate. The resulting intervals can be far too wide, or can even exclude the observed success rate.

Root cause: _binomial_cdf accumulates the pmf by recurrence starting from term = (1.0 - p) ** n. When n * log(1 - p) < ~-745, that starting term is 0.0 in float64, so every later term is also 0.0 and the function returns 0.0 instead of the true CDF. For example, _binomial_cdf(1000, 2000, 0.5) returns 0.0, but the true value is ≈ 0.509. The docstring says the recurrence avoids overflow at large n, but the starting term underflows instead. The bisection in _clopper_pearson_bound then follows the wrong CDF, which corrupts both the lower bound (via 1 - cdf(k - 1)) and the upper bound.

Code example

from transformer_lens.utilities import success_rate_ci
from transformer_lens.utilities.statistics import _binomial_cdf

print(_binomial_cdf(1000, 2000, 0.5))  # 0.0  (true value ~0.509)

for k, n in [(150, 150), (200, 200), (199, 200), (297, 300), (1000, 1000), (1000, 2000), (1990, 2000)]:
    r = success_rate_ci([True] * k + [False] * (n - k))
    print(f"{k}/{n}: rate={r.point_estimate:.4f}  CI=[{r.ci_low:.4f}, {r.ci_high:.4f}]")

Output on current dev (4a92ca50), compared with the exact Clopper-Pearson bounds from scipy.stats.beta.ppf:

successes / trials observed rate success_rate_ci (95%) correct (95%)
150 / 150 1.0000 [0.9757, 1.0000] [0.9757, 1.0000] ✅
200 / 200 1.0000 [0.9759, 1.0000] [0.9817, 1.0000] ❌
199 / 200 0.9950 [0.9725, 0.9759] [0.9725, 0.9999] ❌ (excludes the observed rate)
297 / 300 0.9900 [0.9166, 0.9166] [0.9711, 0.9979] ❌ (excludes the observed rate)
1000 / 1000 1.0000 [0.5243, 1.0000] [0.9963, 1.0000] ❌
1000 / 2000 0.5000 [0.3103, 0.3110] [0.4779, 0.5221] ❌ (excludes the observed rate)
1990 / 2000 0.9950 [0.3103, 0.3110] [0.9908, 0.9976] ❌ (excludes the observed rate)

Reference values:

from scipy.stats import beta
lo = beta.ppf(0.025, k, n - k + 1)   # 0.0 when k == 0
hi = beta.ppf(0.975, k + 1, n - k)   # 1.0 when k == n

At a 50% rate, the first wrong result appears at n = 978: the upper bound is 0.5332 where the correct value is 0.5318.

Expected behavior

Exact Clopper-Pearson bounds for any n, matching scipy.stats.beta.ppf. At minimum, the interval should always contain the point estimate.

Possible fix

Compute the CDF in log space, so no individual term underflows before it is summed. For example, sum exp(log_pmf_i - max_log_pmf) with log_pmf_i = lgamma(n+1) - lgamma(i+1) - lgamma(n-i+1) + i*log(p) + (n-i)*log1p(-p), then rescale. I checked this locally by patching _binomial_cdf with such a log-space version:

  • the CDF matched scipy.stats.binom.cdf to within ~1e-12 across n ∈ {10, 200, 2000} and p ∈ {0.01, 0.5, 0.97, 0.999};
  • every interval in the table above matched the scipy reference to within ~2e-13;
  • the docstring example (0/6 gives [0.0, 0.459]) was unchanged.

A unit test comparing success_rate_ci against a few precomputed Clopper-Pearson values at n = 200–2000 would guard against this regressing.

System Info

  • transformer_lens installed from source (uv sync), dev branch at 4a92ca50
  • OS: Linux (x86_64)
  • Python 3.12.14, torch 2.11.0

Additional context

success_rate_ci is public API (transformer_lens.utilities.success_rate_ci), and it is used to report top-1 success rates in the Jacobian lens causal-swap benchmark. Benchmarks with a few hundred trials and a high success rate are exactly the regime where the interval goes wrong. The function's docstring recommends it over the bootstrap precisely because it is exact, so users are unlikely to suspect its output.

Checklist
  • I have checked that there is no similar issue in the repo (required)
主要言語
Python
スター
3.9k
フォーク
708
平均マージ
2日 2時間
マージ済み PR（30日）
79

環境構築

Codespaces で開く

このプロジェクトの開発コンテナを、あなたの GitHub アカウントでブラウザ上に起動します。

  • Dockerfile・Docker Compose ファイルなし
  • プルリクエストのテンプレートあり
  • コントリビューションガイドなし

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

TransformerLensOrg/TransformerLens のほかの issue

TransformerLensOrg/TransformerLens の issue をすべて見る

似ている issue

Python の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。