Per-device FP8 storage format breaks shared CPU weights on mixed-capability GPUs

Aperta
  • Qualcosa non va?
#10,142 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 2 giorni

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
52/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
python, pytorch
Ambito
backend, machine-learning

Direzione di ricerca

Start by reading ModelLoader._cache_key_variant() and the T5 loader implementation in tests/backend/model_manager/load/test_t5_single_file_loader.py, then inspect the listed loaders: mistral_encoder.py, qwen_image.py, krea2.py, ideogram4.py, and load_default.py. Add format-specific cache variants using one memoized capability decision per load, and test adoption in both orders across devices with different FP8 support; done means each device gets a compatible stored format while compatible formats can share weights.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Summary

Some loaders pick how they store weights based on the device they load for: FP8 kept, or folded into bf16/fp32. The per-device model caches still share one canonical CPU copy per cache key (SharedCpuWeightsStore, adopted in ModelLoader._try_adopt_shared_weights). On a machine whose GPUs differ in FP8 support, the second device takes over the first device's format without running the loader again:

  • A device that cannot run float8 adopts FP8 weights and fails on its first forward ("not implemented for Float8").
  • In the other load order, an FP8-capable device adopts the folded copy and loses the VRAM saving it would have had.

PR #9799 fixes this for the ComfyUI single-file T5 loader with a new hook, ModelLoader._cache_key_variant(). A loader returns a suffix (e.g. fp8) for the format it builds, so only devices that build the same format share a canonical copy. The decision is made once per load so the key and the stored format cannot disagree.

Affected loaders on main

Each of these decides the stored format per device but uses the plain cache key:

  • invokeai/backend/model_manager/load/model_loaders/mistral_encoder.py: keep_fp8 = should_keep_fp8_weights(target_device) or _device_supports_fp8_storage(target_device, logger)
  • invokeai/backend/model_manager/load/model_loaders/qwen_image.py: keep_fp8 = bool(fp8_layers) and (should_keep_fp8_weights(target_device) or use_fp8_storage)
  • invokeai/backend/model_manager/load/model_loaders/krea2.py: keep_matmul_fp8 = should_keep_fp8_weights(target_device)
  • invokeai/backend/model_manager/load/model_loaders/ideogram4.py: if marked and should_keep_fp8_weights(self._torch_device)
  • FP8 Storage in the base loader (load_default.py, _fp8_storage_asked_for / _device_supports_fp8_storage): the adoption path keeps fp8 cast hooks carried by the shell, so a device without float8 support can adopt hooked float8 weights.

Proposed fix

After #9799 merges, have each of these loaders implement _cache_key_variant(), returning the format it will build, and memoize the capability decision per load the way the T5 loader does. Each needs a test that runs adoption in both load orders across two devices with different FP8 support (#9799 has a template in tests/backend/model_manager/load/test_t5_single_file_loader.py).

Side note: working_memory_calibration_fits.measured_models splits keys at the last : and expects a submodel name there. A suffixed key would be misattributed if a model with a variant ever gets a working-memory estimator. That is not reachable today, but worth handling together.

Impact

This only happens with several GPUs of different FP8 capability in one process. Single-GPU setups are not affected.

Lingua principale
TypeScript
Stelle
29k
Fork
3.1k
Merge medio
4g 2h
PR unite (30g)
208

Preparare l'ambiente

  • Nessun Dockerfile né file Docker Compose
  • Ha un modello di pull request
  • Nessuna guida per i contributori

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di invoke-ai/InvokeAI

Tutte le issue di invoke-ai/InvokeAI

Issue simili

Altre issue su TypeScript

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.