[Bug Report] `blocks.N.hook_out` never fires on OpenAI GPT because the HF block returns a list

未关闭
  • 哪里不对？
#1,969 2 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 1 天内回复

@ska2704 已经在做这个了。

开始于 2026年11月3日。

评估

难度
3/5
预计耗时
半天
新手友好度
72/100
Issue 类型
缺陷
描述清晰度
描述清楚
活跃度
活跃
技术栈
huggingface, python, pytorch
领域
machine-learning

调研方向

Start in _apply_output_hook in transformer_lens/model_bridge/generalized_components/block.py, which handles only tuple and bare-tensor outputs. Add a list branch that hooks the first element and returns a list of the same length, then check _stop_on_block_exit for the special case that can be removed afterwards. Reproduce with the snippet in the issue, then add a test for the OpenAI GPT adapter. Done when blocks.0.hook_out is cached and a zeroing hook on the last block's hook_out changes the logits.

由索引模型根据 Issue 内容生成。

描述

bug complexity-simple TransformerBridge

Describe the bug

On openai-community/openai-gpt, every blocks.N.hook_out hook point is dead. It is absent from run_with_cache output and a hook added to it with run_with_hooks or add_hook is never called. hook_in works, the block is wired correctly, only the output side is skipped.

The cause is in BlockBridge._apply_output_hook (transformer_lens/model_bridge/generalized_components/block.py). It hooks the first element of a tuple output, or a bare tensor, and returns anything else untouched. HF's OpenAIGPT block returns outputs = [h] + attn_outputs[1:], a list (transformers/models/openai/modeling_openai.py), so it falls through the untouched branch and hook_out is never called.

Consequences: hook_resid_post is missing from the cache on this model, residual-stream patching at block outputs silently does nothing, and analysis tools that read blocks.N.hook_out get a KeyError. #1964 ran into this and handles the list in its exit hook so that stop_at_layer=n_layers still returns the residual, but that leaves an edit at blocks.11.hook_out reflected in the returned residual on GPT-2 and silently dropped on OpenAI GPT.

Code example

import torch
from transformer_lens.model_bridge import TransformerBridge

bridge = TransformerBridge.boot_transformers("openai-community/openai-gpt", device="cpu", dtype=torch.float32)
tokens = bridge.tokenizer("the capital of france is", return_tensors="pt").input_ids

_, cache = bridge.run_with_cache(tokens)
print("blocks.0.hook_in in cache: ", "blocks.0.hook_in" in cache)
print("blocks.0.hook_out in cache:", "blocks.0.hook_out" in cache)

fired = []
bridge.run_with_hooks(tokens, fwd_hooks=[("blocks.0.hook_out", lambda t, hook: fired.append(hook.name))])
print("hook_out calls:", len(fired))

Output:

blocks.0.hook_in in cache:  True
blocks.0.hook_out in cache: False
hook_out calls: 0

Expected: hook_out in the cache and the hook called once per forward, as on every other decoder adapter.

System Info

  • transformer_lens installed from source at dev (a7b3213ac)
  • macOS
  • Python 3.12, torch 2.11.0, transformers 5.13.0

Additional context

Suggested fix: give _apply_output_hook a list branch mirroring the tuple one, returning a list of the same length so the HF parent's outputs[0] indexing keeps working. Among the supported adapters, OpenAI GPT is the only one whose block returns a list in transformers 5.13.0 (the other list-returning HF blocks are vision/omni models without adapters), so the blast radius is one adapter. Once this lands, the list special-case in _stop_on_block_exit from #1964 can be dropped.

A regression test on the OpenAI GPT adapter should assert blocks.0.hook_out is cached and that a zeroing hook on the last block's hook_out changes the logits.

Checklist
  • I have checked that there is no similar issue in the repo (required)
主要语言
Python
星标
3.9k
派生
708
平均合并
1 天 9 小时
30 天内合并 PR
84

环境准备

在 Codespaces 中打开

在浏览器里用你自己的 GitHub 账号启动这个项目的开发容器。

  • 没有 Dockerfile 或 Docker Compose 文件
  • 有 Pull Request 模板
  • 没有贡献指南

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

TransformerLensOrg/TransformerLens 的其他 Issue

查看 TransformerLensOrg/TransformerLens 的全部 Issue

相似的 Issue

更多 Python Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。