[Bug Report] `blocks.N.hook_out` never fires on OpenAI GPT because the HF block returns a list

Open
  • Is something wrong?
#1,969 2 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

@ska2704 is already working on this.

Since Nov 3, 2026.

Assessment

Difficulty
3/5
Estimated time
Half a day
Newbie friendliness
72/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
huggingface, python, pytorch
Domain
machine-learning

Research direction

Start in _apply_output_hook in transformer_lens/model_bridge/generalized_components/block.py, which handles only tuple and bare-tensor outputs. Add a list branch that hooks the first element and returns a list of the same length, then check _stop_on_block_exit for the special case that can be removed afterwards. Reproduce with the snippet in the issue, then add a test for the OpenAI GPT adapter. Done when blocks.0.hook_out is cached and a zeroing hook on the last block's hook_out changes the logits.

Written by the indexing model from the issue text.

Description

bug complexity-simple TransformerBridge

Describe the bug

On openai-community/openai-gpt, every blocks.N.hook_out hook point is dead. It is absent from run_with_cache output and a hook added to it with run_with_hooks or add_hook is never called. hook_in works, the block is wired correctly, only the output side is skipped.

The cause is in BlockBridge._apply_output_hook (transformer_lens/model_bridge/generalized_components/block.py). It hooks the first element of a tuple output, or a bare tensor, and returns anything else untouched. HF's OpenAIGPT block returns outputs = [h] + attn_outputs[1:], a list (transformers/models/openai/modeling_openai.py), so it falls through the untouched branch and hook_out is never called.

Consequences: hook_resid_post is missing from the cache on this model, residual-stream patching at block outputs silently does nothing, and analysis tools that read blocks.N.hook_out get a KeyError. #1964 ran into this and handles the list in its exit hook so that stop_at_layer=n_layers still returns the residual, but that leaves an edit at blocks.11.hook_out reflected in the returned residual on GPT-2 and silently dropped on OpenAI GPT.

Code example

import torch
from transformer_lens.model_bridge import TransformerBridge

bridge = TransformerBridge.boot_transformers("openai-community/openai-gpt", device="cpu", dtype=torch.float32)
tokens = bridge.tokenizer("the capital of france is", return_tensors="pt").input_ids

_, cache = bridge.run_with_cache(tokens)
print("blocks.0.hook_in in cache: ", "blocks.0.hook_in" in cache)
print("blocks.0.hook_out in cache:", "blocks.0.hook_out" in cache)

fired = []
bridge.run_with_hooks(tokens, fwd_hooks=[("blocks.0.hook_out", lambda t, hook: fired.append(hook.name))])
print("hook_out calls:", len(fired))

Output:

blocks.0.hook_in in cache:  True
blocks.0.hook_out in cache: False
hook_out calls: 0

Expected: hook_out in the cache and the hook called once per forward, as on every other decoder adapter.

System Info

  • transformer_lens installed from source at dev (a7b3213ac)
  • macOS
  • Python 3.12, torch 2.11.0, transformers 5.13.0

Additional context

Suggested fix: give _apply_output_hook a list branch mirroring the tuple one, returning a list of the same length so the HF parent's outputs[0] indexing keeps working. Among the supported adapters, OpenAI GPT is the only one whose block returns a list in transformers 5.13.0 (the other list-returning HF blocks are vision/omni models without adapters), so the blast radius is one adapter. Once this lands, the list special-case in _stop_on_block_exit from #1964 can be dropped.

A regression test on the OpenAI GPT adapter should assert blocks.0.hook_out is cached and that a zeroing hook on the last block's hook_out changes the logits.

Checklist
  • I have checked that there is no similar issue in the repo (required)
Dominant language
Python
Stars
3.9k
Forks
708
Avg merge
1d 9h
Merged PRs (30d)
84

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

  • No Dockerfile or Docker Compose file
  • Has a pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from TransformerLensOrg/TransformerLens

All issues in TransformerLensOrg/TransformerLens

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.