S-, T- and X-learner bootstrap confidence intervals are not reproducible

Open
  • Is something wrong?
#1,036 1 comment 0 reactions 0 assignees View on GitHub

@su-jin1425 is already working on this.

Since Aug 28, 2026.

  • #1041 by @su-jin1425 — open

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
52/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Quiet
Tech stack
numpy, python, scikit-learn
Domain
machine-learning

Research direction

Start with dependency #1035 and BaseLearner.bootstrap() in base.py:117, then inspect the S-, X- and legacy T-learner call sites listed in slearner.py, xlearner.py and tlearner.py. Done means the affected learner classes expose the seeded parameter, bootstrap intervals are reproducible independently of the global NumPy stream, and clone()/get_params round-trip coverage is present.

Written by the indexing model from the issue text.

Description

bug

Depends on #1035 — that issue establishes the single check_random_state normalization path this one threads through the S- and X-learner families. Landing this first would mean either duplicating that decision or redoing it.

BaseLearner.bootstrap() (base.py:117) takes an optional rng and falls back to the global numpy stream when it is not supplied (base.py:133-137). #1032 fixed this for the R-learner (#1029). The remaining callers still pass no rng, so on master the R-learner is the only meta-learner whose bootstrap confidence intervals are reproducible:

  • slearner.py:215, :308
  • xlearner.py:305, :422
  • tlearner.py:297, :395 — the legacy loop only. T-learner's fit(store_bootstraps=True) → fit_bootstrap_ensemble() path from #886 is already seeded, and that is what predict(return_ci=True) uses.

DR-learner is unaffected: it threads an explicit per-call seed.

So BaseSRegressor(...).estimate_ate(..., bootstrap_ci=True) returns different intervals on every call, and unrelated consumption of the global numpy stream between calls changes the result. There is currently no way for a caller to pin them.

Note this is not purely a threading fix: grep -c random_state is 0 for both slearner.py and xlearner.py, so they have no random_state parameter to thread. Adding one is a backward-compatible API addition, which is why #1029 scoped itself to the R-learner.

Since #912 these are BaseEstimators, so the new parameter must be stored verbatim in __init__ per the scikit-learn convention, and each affected class wants clone() / get_params round-trip coverage. That is seven classes across two families: BaseSLearner, BaseSRegressor, BaseSClassifier, LRSRegressor (which constructs StatsmodelsOLS for the caller and needs care), plus BaseXLearner, BaseXRegressor, BaseXClassifier.

Suggested order: land #1035 first, then add random_state to BaseSLearner and BaseXLearner and pass rng at the six call sites.

Observed on master (477cd0a).

🤖 Generated with Claude Code

Dominant language
Python
Stars
6k
Forks
892
Avg merge
11h 53m
Merged PRs (30d)
1

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from uber/causalml

All issues in uber/causalml

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.