Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

[Bug Report] `boot_transformers(hf_model=...)` without `device=` can record the wrong `cfg.device`

Abierto
  • ¿Algo no está bien?
#1,945 1 comentario 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
3/5
Tiempo estimado
1-2 días
Aptitud para principiantes
56/100
Tipo de issue
Error
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
python, pytorch
Área
backend, machine-learning

Línea de trabajo

Start in transformer_lens/model_bridge/sources/transformers/source.py at the device selection and pre-loaded model fallback noted in the report. Compare that flow with build_bridge_from_module, which takes cfg.device from the model. Reproduce with a pre-loaded model on a different device and check bridge calls, run_with_hooks, and generate; done when cfg.device matches the weights' device and those calls no longer fail.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Describe the bug

Without device=, TransformerBridge.boot_transformers(..., hf_model=model) keeps cfg.device at get_device() unless the model has an hf_device_map, but never moves the model. When the weights are on another device, calls that move their inputs to cfg.device fail, including bridge("text"), run_with_hooks on text and generate. run_with_cache with a single-string input works, because it moves the tokenized inputs to the weights' device.

Code example

from transformers import AutoModelForCausalLM
from transformer_lens.model_bridge import TransformerBridge

hf_model = AutoModelForCausalLM.from_pretrained("gpt2").to("mps")
bridge = TransformerBridge.boot_transformers("gpt2", hf_model=hf_model)
print(bridge.cfg.device)  # cpu
bridge("Hello world")
# RuntimeError: Placeholder storage has not been allocated on MPS device!

Loading with device_map="auto" instead of .to("mps") fails the same way, because a single-device load sets no hf_device_map. A model left on the CPU, where from_pretrained puts it by default, fails with TRANSFORMERLENS_ALLOW_MPS=1 (cfg.device is mps). On a CUDA machine, get_device() returns cuda, so the same CPU model gets cfg.device cuda.

System Info

  • Source checkout installed with uv, dev at 34c5d77f.
  • macOS 27.2, Apple M5; Python 3.12.14; PyTorch 2.11.0; transformers 5.13.0; CPU and MPS.

Additional context

The get_device() default applies whenever no device map resolves (transformer_lens/model_bridge/sources/transformers/source.py:261-264), which is always the case with hf_model=, since device_map, n_devices and max_memory are rejected alongside it. The fallback for a pre-loaded model without an hf_device_map, which reads its first parameter (:414-419), only runs while cfg.device is None, which it never is here. build_bridge_from_module already takes cfg.device from the model.

Proposed fix: call get_device() only when boot_transformers loads the model itself, so that cfg.device stays None for hf_model without device= until it is read from the model after the load. An explicit device= keeps its current behavior: it is recorded but not applied to a pre-loaded model.

The MODEL_PATH cells in demos/LLaMA.ipynb and demos/LLaMA_GPU_Quantized.ipynb load the model on the CPU and pass device=device, so on CUDA or MPS their generate call gets inputs on the accelerator and weights on the CPU. Should boot_transformers move a pre-loaded model to an explicit device=?

Checklist
  • I have checked that there is no similar issue in the repo.
Lenguaje dominante
Python
Estrellas
3.9k
Forks
708
Merge medio
2 d 3 h
PR fusionados (30 d)
88

Preparar el entorno

Abrir en Codespaces

Inicia el contenedor de desarrollo del proyecto en tu navegador, con tu propia cuenta de GitHub.

  • Sin Dockerfile ni archivo de Docker Compose
  • Tiene una plantilla de pull request
  • Sin guía de contribución

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de TransformerLensOrg/TransformerLens

Todos los issues de TransformerLensOrg/TransformerLens

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.