Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

[Bug Report] `success_rate_ci` returns wrong Clopper-Pearson bounds at high success rates or large n (binomial CDF underflows)

Chiusa
  • Qualcosa non va?
#1,926 2 commenti 0 reazioni 1 assegnatario Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

@bercedev ci sta già lavorando.

Dal 21/10/2026.

  • #1931 di @bercedev — integrata

Valutazione

Questa issue non è ancora stata valutata.

Descrizione

bug complexity-moderate good first issue help wanted TransformerBridge

Describe the bug

transformer_lens.utilities.success_rate_ci silently returns incorrect "exact" Clopper-Pearson intervals once (1 - p) ** n underflows to 0.0 inside _binomial_cdf (transformer_lens/utilities/statistics.py). This happens with as few as 200 trials when the success rate is high, and at around n ≈ 980 for a 50% rate. The resulting intervals can be far too wide, or can even exclude the observed success rate.

Root cause: _binomial_cdf accumulates the pmf by recurrence starting from term = (1.0 - p) ** n. When n * log(1 - p) < ~-745, that starting term is 0.0 in float64, so every later term is also 0.0 and the function returns 0.0 instead of the true CDF. For example, _binomial_cdf(1000, 2000, 0.5) returns 0.0, but the true value is ≈ 0.509. The docstring says the recurrence avoids overflow at large n, but the starting term underflows instead. The bisection in _clopper_pearson_bound then follows the wrong CDF, which corrupts both the lower bound (via 1 - cdf(k - 1)) and the upper bound.

Code example

from transformer_lens.utilities import success_rate_ci
from transformer_lens.utilities.statistics import _binomial_cdf

print(_binomial_cdf(1000, 2000, 0.5))  # 0.0  (true value ~0.509)

for k, n in [(150, 150), (200, 200), (199, 200), (297, 300), (1000, 1000), (1000, 2000), (1990, 2000)]:
    r = success_rate_ci([True] * k + [False] * (n - k))
    print(f"{k}/{n}: rate={r.point_estimate:.4f}  CI=[{r.ci_low:.4f}, {r.ci_high:.4f}]")

Output on current dev (4a92ca50), compared with the exact Clopper-Pearson bounds from scipy.stats.beta.ppf:

successes / trials observed rate success_rate_ci (95%) correct (95%)
150 / 150 1.0000 [0.9757, 1.0000] [0.9757, 1.0000] ✅
200 / 200 1.0000 [0.9759, 1.0000] [0.9817, 1.0000] ❌
199 / 200 0.9950 [0.9725, 0.9759] [0.9725, 0.9999] ❌ (excludes the observed rate)
297 / 300 0.9900 [0.9166, 0.9166] [0.9711, 0.9979] ❌ (excludes the observed rate)
1000 / 1000 1.0000 [0.5243, 1.0000] [0.9963, 1.0000] ❌
1000 / 2000 0.5000 [0.3103, 0.3110] [0.4779, 0.5221] ❌ (excludes the observed rate)
1990 / 2000 0.9950 [0.3103, 0.3110] [0.9908, 0.9976] ❌ (excludes the observed rate)

Reference values:

from scipy.stats import beta
lo = beta.ppf(0.025, k, n - k + 1)   # 0.0 when k == 0
hi = beta.ppf(0.975, k + 1, n - k)   # 1.0 when k == n

At a 50% rate, the first wrong result appears at n = 978: the upper bound is 0.5332 where the correct value is 0.5318.

Expected behavior

Exact Clopper-Pearson bounds for any n, matching scipy.stats.beta.ppf. At minimum, the interval should always contain the point estimate.

Possible fix

Compute the CDF in log space, so no individual term underflows before it is summed. For example, sum exp(log_pmf_i - max_log_pmf) with log_pmf_i = lgamma(n+1) - lgamma(i+1) - lgamma(n-i+1) + i*log(p) + (n-i)*log1p(-p), then rescale. I checked this locally by patching _binomial_cdf with such a log-space version:

  • the CDF matched scipy.stats.binom.cdf to within ~1e-12 across n ∈ {10, 200, 2000} and p ∈ {0.01, 0.5, 0.97, 0.999};
  • every interval in the table above matched the scipy reference to within ~2e-13;
  • the docstring example (0/6 gives [0.0, 0.459]) was unchanged.

A unit test comparing success_rate_ci against a few precomputed Clopper-Pearson values at n = 200–2000 would guard against this regressing.

System Info

  • transformer_lens installed from source (uv sync), dev branch at 4a92ca50
  • OS: Linux (x86_64)
  • Python 3.12.14, torch 2.11.0

Additional context

success_rate_ci is public API (transformer_lens.utilities.success_rate_ci), and it is used to report top-1 success rates in the Jacobian lens causal-swap benchmark. Benchmarks with a few hundred trials and a high success rate are exactly the regime where the interval goes wrong. The function's docstring recommends it over the bootstrap precisely because it is exact, so users are unlikely to suspect its output.

Checklist
  • I have checked that there is no similar issue in the repo (required)
Lingua principale
Python
Stelle
3.9k
Fork
708
Merge medio
2g 3h
PR unite (30g)
88

Preparare l'ambiente

Apri in Codespaces

Avvia il container di sviluppo del progetto nel browser, con il tuo account GitHub.

  • Nessun Dockerfile né file Docker Compose
  • Ha un modello di pull request
  • Nessuna guida per i contributori

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di TransformerLensOrg/TransformerLens

Tutte le issue di TransformerLensOrg/TransformerLens

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.