[REQUEST] Ulysses support for VLMs

Aperta
  • Qualcosa non va?
#7,834 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 2 giorni

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
25/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Ferma
Stack tecnologico
huggingface, python, pytorch
Ambito
distributed-systems, machine-learning

Direzione di ricerca

Inizia in deepspeed/runtime/sequence_parallel/ulysses_sp.py, in UlyssesSPAttentionHF.register_with_transformers, e traccia il modo in cui ALL_ATTENTION_FUNCTIONS viene sovrascritto. Confronta i percorsi di configurazione del modello linguistico e del vision encoder in Qwen/Qwen3-VL-8B-Instruct, quindi determina come fare in modo che la registrazione abbia come target solo il modello linguistico. Il lavoro è completato quando il VLM usa Ulysses attention per il proprio modello linguistico senza applicarla al vision encoder; l’issue richiede indicazioni dai maintainer sull’approccio esatto al caricamento.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

enhancement

Feature description
UlyssesSPAttentionHF.register_with_transformers assumes that a text-only model will be used. I am hoping to use it with a VLM (for example, Qwen/Qwen3-VL-8B-Instruct.)

UlyssesSPAttentionHF.register_with_transformers assumes certain keys in the config, for example: attn_head_count=hf_model_config.num_attention_heads,. However for Qwen3VL, the correct key would be hf_model_config.text_config.num_attention_heads. This would be easy enough to change to make it specific to the language model (or the vision encoder, as necessary.)

However the broader problem is that this new attention implementation is architecture dependent (ie. depends on the number of heads) and is used to overwrite the global attention implementation:

# So instead we hack `ALL_ATTENTION_FUNCTIONS` to override all existing keys with our implementation, since it only gets used at the point of calling the attention and that's what we want, all other code branches relying on the original core `attn_implementation` will still be executed. This is what we called "Being John Malkovich"
for key in ALL_ATTENTION_FUNCTIONS.keys():
    ALL_ATTENTION_FUNCTIONS[key] = uattn_wrapper

This enables loading the model with no changes - ie. AutoModel.from_pretrained. However the vision encoder has a different architecture with a different number of heads. When it is loaded, it will have the same sequence parallel attention as the language model, which would be incorrect. Further, it is likely unnecessary to implement sequence parallel attention for the vision encoder because it is much smaller than the language model. But because the global attention implementation is overwritten with the Ulysses attention, changing the attention implementation for the language model will change it for the vision encoder as well.

Describe the solution you'd like
The ability to use UlyssesSPAttentionHF.register_with_transformers to enable sequence parallel for the language model, but not the vision encoder, of a VLM.

Describe alternatives you've considered
For selecting the correct config keys, an additional parameter could be added to UlyssesSPAttentionHF.register_with_transformers that allows for specifying the config key that contains the architectural details for the new attention function.

For the global overwriting of the attention implementations, it's less clear to me. Maybe the vision encoder should be loaded before the global overwriting, and then the language model is loaded after? And then the two are connected back together? However this would mean messing with the model loading.

I am happy to make the necessary changes and submit a PR, but direction on the best approach to solving these issues would be appreciated.

Lingua principale
Python
Stelle
43.3k
Fork
5.1k
Merge medio
5g 40m
PR unite (30g)
75

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di deepspeedai/DeepSpeed

Tutte le issue di deepspeedai/DeepSpeed

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.