Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

[Bug Report] `boot_transformers(hf_model=...)` without `device=` can record the wrong `cfg.device`

Aperta
  • Qualcosa non va?
#1,945 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
3/5
Tempo stimato
1-2 giorni
Idoneità per principianti
56/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
python, pytorch
Ambito
backend, machine-learning

Direzione di ricerca

Start in transformer_lens/model_bridge/sources/transformers/source.py at the device selection and pre-loaded model fallback noted in the report. Compare that flow with build_bridge_from_module, which takes cfg.device from the model. Reproduce with a pre-loaded model on a different device and check bridge calls, run_with_hooks, and generate; done when cfg.device matches the weights' device and those calls no longer fail.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Describe the bug

Without device=, TransformerBridge.boot_transformers(..., hf_model=model) keeps cfg.device at get_device() unless the model has an hf_device_map, but never moves the model. When the weights are on another device, calls that move their inputs to cfg.device fail, including bridge("text"), run_with_hooks on text and generate. run_with_cache with a single-string input works, because it moves the tokenized inputs to the weights' device.

Code example

from transformers import AutoModelForCausalLM
from transformer_lens.model_bridge import TransformerBridge

hf_model = AutoModelForCausalLM.from_pretrained("gpt2").to("mps")
bridge = TransformerBridge.boot_transformers("gpt2", hf_model=hf_model)
print(bridge.cfg.device)  # cpu
bridge("Hello world")
# RuntimeError: Placeholder storage has not been allocated on MPS device!

Loading with device_map="auto" instead of .to("mps") fails the same way, because a single-device load sets no hf_device_map. A model left on the CPU, where from_pretrained puts it by default, fails with TRANSFORMERLENS_ALLOW_MPS=1 (cfg.device is mps). On a CUDA machine, get_device() returns cuda, so the same CPU model gets cfg.device cuda.

System Info

  • Source checkout installed with uv, dev at 34c5d77f.
  • macOS 27.2, Apple M5; Python 3.12.14; PyTorch 2.11.0; transformers 5.13.0; CPU and MPS.

Additional context

The get_device() default applies whenever no device map resolves (transformer_lens/model_bridge/sources/transformers/source.py:261-264), which is always the case with hf_model=, since device_map, n_devices and max_memory are rejected alongside it. The fallback for a pre-loaded model without an hf_device_map, which reads its first parameter (:414-419), only runs while cfg.device is None, which it never is here. build_bridge_from_module already takes cfg.device from the model.

Proposed fix: call get_device() only when boot_transformers loads the model itself, so that cfg.device stays None for hf_model without device= until it is read from the model after the load. An explicit device= keeps its current behavior: it is recorded but not applied to a pre-loaded model.

The MODEL_PATH cells in demos/LLaMA.ipynb and demos/LLaMA_GPU_Quantized.ipynb load the model on the CPU and pass device=device, so on CUDA or MPS their generate call gets inputs on the accelerator and weights on the CPU. Should boot_transformers move a pre-loaded model to an explicit device=?

Checklist
  • I have checked that there is no similar issue in the repo.
Lingua principale
Python
Stelle
3.9k
Fork
708
Merge medio
2g 3h
PR unite (30g)
88

Preparare l'ambiente

Apri in Codespaces

Avvia il container di sviluppo del progetto nel browser, con il tuo account GitHub.

  • Nessun Dockerfile né file Docker Compose
  • Ha un modello di pull request
  • Nessuna guida per i contributori

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di TransformerLensOrg/TransformerLens

Tutte le issue di TransformerLensOrg/TransformerLens

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.