Streamed group offloading with `low_cpu_mem_usage=True` re-pins every group on every forward pass

Ouverte
#15,256 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub

Les mainteneurs répondent en général sous 1 jour

Personne n'a encore pris cette issue.

Évaluation

Difficulté
4/5
Temps estimé
3-5 jours
Accessibilité débutants
42/100
Type d'issue
Bug
Clarté
Plutôt claire
Activité
Active
Stack technique
python, pytorch

Piste de recherche

Start with group_offloading.py, focusing on ModuleGroup._pinned_memory_tensors() and _apply_group_offloading_block_level; the issue also names tests/hooks/test_group_offloading.py. Trace how streamed groups pin CPU tensors and how unmatched_group handles offloading when low CPU memory usage is enabled. Done means reusable staging buffers avoid repeated pinning, original CPU tensors are restored on offload, and a test covers the behavior; the reported video difference still needs a same-process comparison.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Description

Problem

With streamed group offloading and low_cpu_mem_usage=True:

transformer.enable_group_offload(
    onload_device=torch.device("cuda"),
    offload_device=torch.device("cpu"),
    offload_type="block_level",
    num_blocks_per_group=1,
    use_stream=True,
    record_stream=True,
    low_cpu_mem_usage=True,
)

ModuleGroup._pinned_memory_tensors() calls tensor.pin_memory() on every parameter of the group each time the group is onloaded, and drops the pinned copies right after the transfer. For a large transformer this means allocating and page-locking the full model size in host memory on every denoising step (LTX-2.5's transformer is ~38 GB in BF16, so 8 steps pin ~300 GB in total).

A second, smaller issue: the root unmatched_group built in _apply_group_offloading_block_level is created without low_cpu_mem_usage=config.low_cpu_mem_usage, and because it has stream=None its offload path does .to(offload_device), which allocates fresh CPU copies of those tensors instead of returning to the original (e.g. safetensors-mmapped) storage.

Proposed change
  1. Keep a small reusable pinned staging buffer per stream (two slots, each sized to the largest group, so the current transfer and the prefetched next group never share a slot). Contiguous unpinned CPU tensors are copied into a slot view and transferred from there; TorchAO and non-contiguous tensors keep the current path.
  2. When low_cpu_mem_usage=True, remember the original CPU tensors and restore them on offload, and pass low_cpu_mem_usage through to unmatched_group.
Measurements

Lightricks/LTX-2.5-Diffusers, BF16, 512×320, 49 frames, 8 distilled steps, seed 42, prompt pre-encoded; connectors, transformer, VAE, audio VAE and vocoder offloaded with the settings above. AMD Radeon PRO W7800 (32 GB), torch 2.13.0+rocm10.0.0. Mean of two warm pipeline calls after one warmup call:

Warm end-to-end Process RssAnon
main 55.62 s (56.34, 54.89) 10.56 GiB
proposed 49.82 s (49.66, 49.97) 9.18 GiB

The audio output is bitwise identical between the two runs. The decoded video differs by up to 1.8e-2 (values in [0, 1]); the change only alters how identical bytes are staged on the host, so I suspect cross-process convolution algorithm selection in the VAE decode, but I have not isolated it yet and will confirm with a same-process comparison before opening a PR.

Coordination

This is separate from #12747 (configurable pinning of groups). If the approach looks reasonable, I can open a small PR (changes in group_offloading.py plus a test in tests/hooks/test_group_offloading.py).

Langage dominant
Python
Étoiles
34.9k
Forks
7.5k
Merge moyen
3 j 16 h
PR mergées (30 j)
65

Préparer son environnement

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Autres issues de huggingface/diffusers

Toutes les issues de huggingface/diffusers

Issues similaires

Plus d'issues Python

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.