Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

[Proposal] Several positions per example in interchange interventions

Aperta
  • Qualcosa non va?
#1,939 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
35/100
Tipo di issue
Funzionalità
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
python, pytorch
Ambito
machine-learning

Direzione di ricerca

Start with the interchange tools in tools.analysis.interchange and the existing interchange_interventions.md documentation. The proposal specifies aligned [batch, k] positions and mean pooled spans, including padding with -1; add unit coverage for both modes and padding, plus an integration test reproducing the GPT-2 result. Done means both modes work with existing interventions and fit_das, and the docs explain when pooling differs from alignment.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Proposal

Let the interchange tools in tools.analysis.interchange intervene on several positions per example, in two modes:

  • Aligned positions. positions (and fit_das's base_positions/source_positions) may be [batch, k] as well as [batch]. Base position j takes the source's activation at source
    position j, as with pyvene's (batch, k) unit locations.
  • Pooled spans. With pool="mean", each row names a span instead. The intervention acts on
    the span's mean, and base and source spans may differ in length.

[batch] positions keep working exactly as they do now.

Motivation

The tools intervene on one position per example, but a variable is often spread over several, and
then a one-position interchange underestimates it. On GPT-2 small, with subject number in
"The key(s) near the cabinet", patching the whole residual stream after block 4 at the subject
alone flips 37% of held-out pairs, and at the preposition alone 27%. Patching both together flips
100%. A one-position analysis would conclude that the variable is mostly gone from both positions
at that layer, when the two positions carry it jointly.

The same holds for speech. Whisper's encoder emits a frame every 20 ms, so one word spans many
positions, and the number differs by speaker and speaking rate. Probing work on speech compares
single frames, quarter-segment spans and mean-pooled word segments (Pasad et al., 2024). The causal
counterpart needs both aligned positions and pooled spans.

Pitch

Aligned mode. Interventions keep their contract of mapping [n, d_model] base rows and
source rows to new base rows. Each (example, position) pair is one row:

  • collect_activations returns [batch, k, d_model];
  • run_with_intervention flattens the selected positions into rows, applies the intervention and
    writes them back.

So SubspaceInterchange, NormMatchedPlacebo, every basis and fit_das work unchanged, with one
basis shared by the k positions. -1 pads shorter rows, so pairs in a batch can have different
k; base and source pad the same slots.

Pooled mode. Rows of positions name a span per example (padded with -1), and
collect_activations returns the span means, [batch, d_model]. The intervention maps the base's
span mean and the source's to a new mean. Every base position in the span then moves by the same
amount: the new mean minus the base's mean. For SubspaceInterchange, this gives the span's mean
the source's coordinates inside the subspace, and each position keeps its own offset from the
mean. No alignment is needed, so base and source spans can differ in length.

Docs: a "Several positions" section in interchange_interventions.md with the example below.
Tests: unit tests for both modes and the padding, and an integration test reproducing the GPT-2
result.

Alternatives
  • Separate calls per position. They can't express a joint intervention, because each call
    patches one position, and the hook must fire once per pass.
  • A concatenated basis. By default, pyvene's rotation (DAS) intervention flattens
    (batch, k, d) into (batch, k·d) (unless keep_last_dim is set), so the learned subspace
    spans the concatenated positions. The basis then grows with k, k must be fixed, and the
    difference-in-means, probe and neuron bases and their placebos no longer apply. A basis shared
    across positions keeps all of them.
  • Boolean masks instead of indices. A mask can't say which base position takes which source
    position, so it suits pooling but not alignment.
Additional context

The GPT-2 numbers come from 16 nouns, 4 sentence frames, and 12 splits that fit on one frame and
evaluate 32 pairs in another. Each entry is the mean interchange accuracy over the splits, with
the range below. "Whole" patches the whole residual, the ceiling. "DiM" uses a rank-1
difference-in-means basis fit on the fit frame, shared by both positions in the aligned mode.

layer basis subject preposition both, aligned both, pooled
3 whole 0.96 0.00 1.00 0.98
4 whole 0.37 (0.19–0.69) 0.27 (0.00–0.44) 1.00 (1.00–1.00) 0.99
4 DiM 0.31 0.13 0.95 (0.91–0.97) 0.92
5 whole 0.32 (0.19–0.59) 0.08 1.00 (1.00–1.00) 0.92
6 whole 0.27 0.01 0.94 0.86

Pooling is a different intervention, not a cheaper version of alignment. At layer 2, where the
variable sits on the subject alone, pooling over both positions halves the subject's shift and
reaches 0.62 against 1.00. The docs would say so.

References:
Wu, Geiger, Arora, Huang, Wang, Goodman, Manning and Potts (2024), "pyvene: A Library for Understanding and Improving PyTorch Models via Interventions", NAACL (System Demonstrations);
Pasad, Chien, Settle and Livescu (2024), "What Do Self-Supervised Speech Models Know About Words?", TACL 12;
Geiger, Wu, Potts, Icard and Goodman (2024), "Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representations", CLeaR.

Checklist
  • I have checked that there is no similar issue in the repo (required)
Lingua principale
Python
Stelle
3.9k
Fork
708
Merge medio
2g 3h
PR unite (30g)
88

Preparare l'ambiente

Apri in Codespaces

Avvia il container di sviluppo del progetto nel browser, con il tuo account GitHub.

  • Nessun Dockerfile né file Docker Compose
  • Ha un modello di pull request
  • Nessuna guida per i contributori

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di TransformerLensOrg/TransformerLens

Tutte le issue di TransformerLensOrg/TransformerLens

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.