Hacktoberfest 2026：メンテナが10月に向けて印を付けた、オープンで初心者向けの issue。 Hacktoberfest の issue を見る

[Proposal] Neuron bases for interchange interventions

オープン
  • 何か問題がありますか？
#1,917 コメント 1 件 リアクション 0 件 担当者 1 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

@dipo101 がすでに取り組んでいます。

2026年10月20日 から。

評価

この issue はまだ評価されていません。

説明

Proposal

Add neuron_basis(d_mlp, neurons) and random_neuron_basis(d_mlp, k, seed) to tools.analysis.interchange. A chosen set of MLP neurons, such as the features a sparse probe selects, can then be interchanged with SubspaceInterchange and compared with the module's other bases and controls.

Motivation

fit_sparse_probe finds the neurons that best separate a variable, but, as its docs say, that "does not establish causal model use". Interchanging those neurons is the causal test, and can be placed beside difference-in-means and DAS at the same hook, with random neurons as the placebo. IIUC, a random direction is the wrong placebo for a claim about neurons, because it spreads over every neuron. The question is a timely one. The recent Goodfire essay by Nick Cammarata argues that MLP neurons in trained models are mostly clean units, against the superposition picture, and this would make that claim testable on any task.

Pitch

Fit a sparse probe on blocks.N.mlp.hook_post activations, pass result.selected_features to neuron_basis, and evaluate with intervened_logits on held-out pairs. random_neuron_basis draws k neurons for the placebo, either directly or through NormMatchedPlacebo. The helpers only build checked coordinate bases (indices in range, no repeats, seeding local as in random_basis), so the intervention and its evaluation stay exactly as they are.

Alternatives

Users can write torch.eye(d_mlp)[:, idx] themselves, but then the placebo and the link to sparse probing are left to them, as the trained controls were before #1894. Neuron-level patching in patching.py would cover one neuron at a time, but not sets of neurons or a comparison with learned subspaces.

Additional context

On GPT-2 small, with subject number at the subject token, the comparison is concrete. There were 15 runs per layer. 5 seeds for DAS, the sparse probe and the random neurons, each on 3 splits that fit on one sentence frame and evaluate on 32 pairs in another.

At blocks.0.mlp.hook_post, the single neuron a 1-sparse probe selects reaches an interchange accuracy of 1.00 in every run, matching difference-in-means and DAS. Single random neurons reach 0.00, and sets of 8 random neurons average 0.04.

At layer 2, difference-in-means and the 8- and 32-neuron selections reach 0.00, and the single selected neuron reaches 0.44–0.50, while DAS reaches 0.78–1.00. At layer 1, everything but DAS reaches 0.00, and DAS ranges from 0.00 to 0.97 across seeds. DAS finding effects where the model's own directions and neurons do not is consistent with the illusion Makelov et al. (2024) describe for subspace patching of MLP activations, and the neuron baseline is what makes it visible.

References:
Cammarata, "Neural Networks Are Already Clean", Goodfire (undated web essay), https://already-clean.goodfire.com/

Gurnee et al. (2023), "Finding Neurons in a Haystack: Case Studies with Sparse Probing", arXiv:2305.01610

Elhage et al. (2022), "Toy Models of Superposition", Transformer Circuits Thread https://arxiv.org/abs/2209.10652

Makelov, Lange, Geiger and Nanda (2024), "Is This the Subspace You Are Looking for? An Interpretability Illusion for Subspace Activation Patching", https://arxiv.org/abs/2311.17030.

Checklist
  • I have checked that there is no similar issue in the repo (required)
主要言語
Python
スター
3.9k
フォーク
708
平均マージ
2日 1時間
マージ済み PR（30日）
78

環境構築

Codespaces で開く

このプロジェクトの開発コンテナを、あなたの GitHub アカウントでブラウザ上に起動します。

  • Dockerfile・Docker Compose ファイルなし
  • プルリクエストのテンプレートあり
  • コントリビューションガイドなし

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

TransformerLensOrg/TransformerLens のほかの issue

TransformerLensOrg/TransformerLens の issue をすべて見る

似ている issue

Python の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。