Per-device FP8 storage format breaks shared CPU weights on mixed-capability GPUs

Abierto
  • ¿Algo no está bien?
#10,142 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 2 días

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
52/100
Tipo de issue
Error
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
python, pytorch
Área
backend, machine-learning

Línea de trabajo

Start by reading ModelLoader._cache_key_variant() and the T5 loader implementation in tests/backend/model_manager/load/test_t5_single_file_loader.py, then inspect the listed loaders: mistral_encoder.py, qwen_image.py, krea2.py, ideogram4.py, and load_default.py. Add format-specific cache variants using one memoized capability decision per load, and test adoption in both orders across devices with different FP8 support; done means each device gets a compatible stored format while compatible formats can share weights.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Summary

Some loaders pick how they store weights based on the device they load for: FP8 kept, or folded into bf16/fp32. The per-device model caches still share one canonical CPU copy per cache key (SharedCpuWeightsStore, adopted in ModelLoader._try_adopt_shared_weights). On a machine whose GPUs differ in FP8 support, the second device takes over the first device's format without running the loader again:

  • A device that cannot run float8 adopts FP8 weights and fails on its first forward ("not implemented for Float8").
  • In the other load order, an FP8-capable device adopts the folded copy and loses the VRAM saving it would have had.

PR #9799 fixes this for the ComfyUI single-file T5 loader with a new hook, ModelLoader._cache_key_variant(). A loader returns a suffix (e.g. fp8) for the format it builds, so only devices that build the same format share a canonical copy. The decision is made once per load so the key and the stored format cannot disagree.

Affected loaders on main

Each of these decides the stored format per device but uses the plain cache key:

  • invokeai/backend/model_manager/load/model_loaders/mistral_encoder.py: keep_fp8 = should_keep_fp8_weights(target_device) or _device_supports_fp8_storage(target_device, logger)
  • invokeai/backend/model_manager/load/model_loaders/qwen_image.py: keep_fp8 = bool(fp8_layers) and (should_keep_fp8_weights(target_device) or use_fp8_storage)
  • invokeai/backend/model_manager/load/model_loaders/krea2.py: keep_matmul_fp8 = should_keep_fp8_weights(target_device)
  • invokeai/backend/model_manager/load/model_loaders/ideogram4.py: if marked and should_keep_fp8_weights(self._torch_device)
  • FP8 Storage in the base loader (load_default.py, _fp8_storage_asked_for / _device_supports_fp8_storage): the adoption path keeps fp8 cast hooks carried by the shell, so a device without float8 support can adopt hooked float8 weights.

Proposed fix

After #9799 merges, have each of these loaders implement _cache_key_variant(), returning the format it will build, and memoize the capability decision per load the way the T5 loader does. Each needs a test that runs adoption in both load orders across two devices with different FP8 support (#9799 has a template in tests/backend/model_manager/load/test_t5_single_file_loader.py).

Side note: working_memory_calibration_fits.measured_models splits keys at the last : and expects a submodel name there. A suffixed key would be misattributed if a model with a variant ever gets a working-memory estimator. That is not reachable today, but worth handling together.

Impact

This only happens with several GPUs of different FP8 capability in one process. Single-GPU setups are not affected.

Lenguaje dominante
TypeScript
Estrellas
29k
Forks
3.1k
Merge medio
4 d 2 h
PR fusionados (30 d)
208

Preparar el entorno

  • Sin Dockerfile ni archivo de Docker Compose
  • Tiene una plantilla de pull request
  • Sin guía de contribución

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de invoke-ai/InvokeAI

Todos los issues de invoke-ai/InvokeAI

Issues similares

Más issues de TypeScript

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.