Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

[Proposal] Whisper adapter for TransformerBridge

Abierto
  • ¿Algo no está bien?
#1,938 1 comentario 0 reacciones 1 asignado Ver en GitHub

Los mantenedores suelen responder en 1 día

@dipo101 ya está trabajando en esto.

Desde el 22/10/2026.

Evaluación

Este issue todavía no se ha evaluado.

Descripción

complexity-moderate new-architecture TransformerBridge
Proposal

Add a WhisperArchitectureAdapter so that WhisperForConditionalGeneration checkpoints load
through TransformerBridge.boot_transformers, with hooks throughout the audio encoder and the
text decoder. That covers openai/whisper-tiny to large-v3, large-v3-turbo and
distil-whisper.

Motivation

#710 asked for Whisper, and its last comment asks for one issue per model. This is that issue.
Today the bridge covers audio in two ways:

  • encoder-only models (HuBERT, wav2vec2, AST);
  • audio-text LMs (Qwen2-Audio, GLM-ASR, Audio Flamingo 3), whose Whisper-style encoders are wrapped opaquely, with hook_in and hook_out only.

No audio encoder-decoder is supported, and nothing exposes the inside of a Whisper encoder. Yet
Whisper is one of the most-studied speech model.

Pitch

The adapter is a BART-family member. Whisper's layers have exactly BART's module names
(self_attn, self_attn_layer_norm, fc1, fc2, final_layer_norm, plus encoder_attn and
encoder_attn_layer_norm in the decoder), with pre-LN and a final layer_norm per stack. So the
adapter subclasses BartFamilyArchitectureAdapter, with these differences:

  • Encoder front end. Hooks on conv1, conv2 and the fixed sinusoidal embed_positions,
    in place of token embeddings. encoder_blocks.{i}.hook_out is then [batch, 1500, d_model],
    one frame per 20 ms of a 30 s window.
  • Unembedding. proj_out, tied to the decoder's embed_tokens, with no final_logits_bias.
  • Asymmetric stacks. large-v3-turbo has 32 encoder and 4 decoder layers, and
    distil-large-v3 has 2 decoder layers, so n_layers follows the decoder, as in Blenderbot.

Changes outside the adapter (the part to agree on first):

  1. Route Whisper to AutoModelForSpeechSeq2Seq in utilities/architectures.py.
  2. Keep the tokenizer for audio encoder-decoders. skip_tokenizer_for_modality drops it for
    audio models, but the decoder reads text. Load WhisperProcessor as bridge.processor.
  3. In forward, the audio branch passes input as input_values, but Whisper takes
    input_features (log-mel, [batch, n_mels, 3000]) plus a decoder input. The adapter would
    declare its audio input name, and an audio encoder-decoder would require decoder_input. The
    automatic shift from input_ids doesn't apply to audio.

Usage: bridge(features, decoder_input=prompt) returns decoder logits under teacher forcing,
and run_with_cache returns encoder and decoder hooks.

Out of scope for the first PR:

  • generate. Whisper's generation adds language detection, timestamps and long-form decoding,
    while patching and probing only need the forward pass.
  • Weight processing such as fold_ln, which is off for the BART family.

Tests: unit tests on a tiny random config; an integration test of logit parity with HF on
openai/whisper-tiny (39M) on synthetic audio; and a verify_models run.

Alternatives
  • Keep Whisper opaque, as in the Qwen2-Audio adapter. That gives no hooks inside the encoder.
  • Use separate tooling, such as WhisperLens, linked in #710. That leaves out the bridge's
    analysis tools (patching, interchange, probes, logit lens).
Additional context

With my multi-position proposal, interchange interventions will work on encoder frames. A
word spans many frames and the number differs by speaker, so frame-level causal work needs either
aligned positions or pooled spans. Once this lands, the same encoder mapping could also give
Qwen2-Audio's audio tower full hooks, in a later PR.

References:
Glazer, Segal-Feldman, Segev, Shamsian, Buchnick, Hetz, Fetaya, Keshet and Navon (2025), "Beyond Transcription: Mechanistic Interpretability in ASR", arXiv:2508.15882;
Radford, Kim, Xu, Brockman, McLeavey and Sutskever (2023), "Robust Speech Recognition via Large-Scale Weak Supervision", ICML.

Checklist
  • I have checked that there is no similar issue in the repo (required)
Lenguaje dominante
Python
Estrellas
3.9k
Forks
708
Merge medio
2 d 3 h
PR fusionados (30 d)
88

Preparar el entorno

Abrir en Codespaces

Inicia el contenedor de desarrollo del proyecto en tu navegador, con tu propia cuenta de GitHub.

  • Sin Dockerfile ni archivo de Docker Compose
  • Tiene una plantilla de pull request
  • Sin guía de contribución

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de TransformerLensOrg/TransformerLens

Todos los issues de TransformerLensOrg/TransformerLens

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.