Hacktoberfest 2026: những issue maintainer đã đánh dấu cho tháng Mười, đang mở và phù hợp người mới. Xem issue Hacktoberfest

[Proposal] Neuron bases for interchange interventions

Đã đóng
  • Có gì không ổn?
#1,917 1 bình luận 0 reaction 1 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

@dipo101 đang làm issue này rồi.

Từ ngày 20/10/2026.

  • #1921 của @dipo101 — đã merge

Đánh giá

Issue này chưa được đánh giá.

Mô tả

Proposal

Add neuron_basis(d_mlp, neurons) and random_neuron_basis(d_mlp, k, seed) to tools.analysis.interchange. A chosen set of MLP neurons, such as the features a sparse probe selects, can then be interchanged with SubspaceInterchange and compared with the module's other bases and controls.

Motivation

fit_sparse_probe finds the neurons that best separate a variable, but, as its docs say, that "does not establish causal model use". Interchanging those neurons is the causal test, and can be placed beside difference-in-means and DAS at the same hook, with random neurons as the placebo. IIUC, a random direction is the wrong placebo for a claim about neurons, because it spreads over every neuron. The question is a timely one. The recent Goodfire essay by Nick Cammarata argues that MLP neurons in trained models are mostly clean units, against the superposition picture, and this would make that claim testable on any task.

Pitch

Fit a sparse probe on blocks.N.mlp.hook_post activations, pass result.selected_features to neuron_basis, and evaluate with intervened_logits on held-out pairs. random_neuron_basis draws k neurons for the placebo, either directly or through NormMatchedPlacebo. The helpers only build checked coordinate bases (indices in range, no repeats, seeding local as in random_basis), so the intervention and its evaluation stay exactly as they are.

Alternatives

Users can write torch.eye(d_mlp)[:, idx] themselves, but then the placebo and the link to sparse probing are left to them, as the trained controls were before #1894. Neuron-level patching in patching.py would cover one neuron at a time, but not sets of neurons or a comparison with learned subspaces.

Additional context

On GPT-2 small, with subject number at the subject token, the comparison is concrete. There were 15 runs per layer. 5 seeds for DAS, the sparse probe and the random neurons, each on 3 splits that fit on one sentence frame and evaluate on 32 pairs in another.

At blocks.0.mlp.hook_post, the single neuron a 1-sparse probe selects reaches an interchange accuracy of 1.00 in every run, matching difference-in-means and DAS. Single random neurons reach 0.00, and sets of 8 random neurons average 0.04.

At layer 2, difference-in-means and the 8- and 32-neuron selections reach 0.00, and the single selected neuron reaches 0.44–0.50, while DAS reaches 0.78–1.00. At layer 1, everything but DAS reaches 0.00, and DAS ranges from 0.00 to 0.97 across seeds. DAS finding effects where the model's own directions and neurons do not is consistent with the illusion Makelov et al. (2024) describe for subspace patching of MLP activations, and the neuron baseline is what makes it visible.

References:
Cammarata, "Neural Networks Are Already Clean", Goodfire (undated web essay), https://already-clean.goodfire.com/

Gurnee et al. (2023), "Finding Neurons in a Haystack: Case Studies with Sparse Probing", arXiv:2305.01610

Elhage et al. (2022), "Toy Models of Superposition", Transformer Circuits Thread https://arxiv.org/abs/2209.10652

Makelov, Lange, Geiger and Nanda (2024), "Is This the Subspace You Are Looking for? An Interpretability Illusion for Subspace Activation Patching", https://arxiv.org/abs/2311.17030.

Checklist
  • I have checked that there is no similar issue in the repo (required)
Ngôn ngữ chính
Python
Star
3.9k
Fork
708
Merge trung bình
1 ngày 21 giờ
Pull request đã merge (30 ngày)
91

Chuẩn bị môi trường

Mở trong Codespaces

Khởi chạy dev container của dự án ngay trên trình duyệt, bằng tài khoản GitHub của bạn.

  • Không có Dockerfile hay tệp Docker Compose
  • Có mẫu pull request
  • Không có hướng dẫn đóng góp

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của TransformerLensOrg/TransformerLens

Tất cả issue của TransformerLensOrg/TransformerLens

Issue tương tự

Thêm issue về Python

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.