Streamed group offloading with `low_cpu_mem_usage=True` re-pins every group on every forward pass

未關閉
#15,256 0 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視

維護者通常 1 天內回覆

還沒有人認領這個 Issue。

評估

難度
4/5
預估耗時
3-5 天
新手友好度
42/100
Issue 類型
缺陷
描述清晰度
基本清楚
活躍度
活躍
技術堆疊
python, pytorch

研究方向

Start with group_offloading.py, focusing on ModuleGroup._pinned_memory_tensors() and _apply_group_offloading_block_level; the issue also names tests/hooks/test_group_offloading.py. Trace how streamed groups pin CPU tensors and how unmatched_group handles offloading when low CPU memory usage is enabled. Done means reusable staging buffers avoid repeated pinning, original CPU tensors are restored on offload, and a test covers the behavior; the reported video difference still needs a same-process comparison.

由索引模型根據 Issue 內容生成。

描述

Problem

With streamed group offloading and low_cpu_mem_usage=True:

transformer.enable_group_offload(
    onload_device=torch.device("cuda"),
    offload_device=torch.device("cpu"),
    offload_type="block_level",
    num_blocks_per_group=1,
    use_stream=True,
    record_stream=True,
    low_cpu_mem_usage=True,
)

ModuleGroup._pinned_memory_tensors() calls tensor.pin_memory() on every parameter of the group each time the group is onloaded, and drops the pinned copies right after the transfer. For a large transformer this means allocating and page-locking the full model size in host memory on every denoising step (LTX-2.5's transformer is ~38 GB in BF16, so 8 steps pin ~300 GB in total).

A second, smaller issue: the root unmatched_group built in _apply_group_offloading_block_level is created without low_cpu_mem_usage=config.low_cpu_mem_usage, and because it has stream=None its offload path does .to(offload_device), which allocates fresh CPU copies of those tensors instead of returning to the original (e.g. safetensors-mmapped) storage.

Proposed change
  1. Keep a small reusable pinned staging buffer per stream (two slots, each sized to the largest group, so the current transfer and the prefetched next group never share a slot). Contiguous unpinned CPU tensors are copied into a slot view and transferred from there; TorchAO and non-contiguous tensors keep the current path.
  2. When low_cpu_mem_usage=True, remember the original CPU tensors and restore them on offload, and pass low_cpu_mem_usage through to unmatched_group.
Measurements

Lightricks/LTX-2.5-Diffusers, BF16, 512×320, 49 frames, 8 distilled steps, seed 42, prompt pre-encoded; connectors, transformer, VAE, audio VAE and vocoder offloaded with the settings above. AMD Radeon PRO W7800 (32 GB), torch 2.13.0+rocm10.0.0. Mean of two warm pipeline calls after one warmup call:

Warm end-to-end Process RssAnon
main 55.62 s (56.34, 54.89) 10.56 GiB
proposed 49.82 s (49.66, 49.97) 9.18 GiB

The audio output is bitwise identical between the two runs. The decoded video differs by up to 1.8e-2 (values in [0, 1]); the change only alters how identical bytes are staged on the host, so I suspect cross-process convolution algorithm selection in the VAE decode, but I have not isolated it yet and will confirm with a same-process comparison before opening a PR.

Coordination

This is separate from #12747 (configurable pinning of groups). If the approach looks reasonable, I can open a small PR (changes in group_offloading.py plus a test in tests/hooks/test_group_offloading.py).

主要語言
Python
星號
34.9k
分支
7.5k
平均合併
3 天 16 小時
30 天內合併 PR
65

環境準備

從這裡開始

  1. 先讀完整個 Issue,再讀專案的貢獻指南。
  2. 在 Issue 下留言說明你要接手 —— 這能避免兩個人做同樣的事。
  3. Fork 儲存庫,在一個分支上完成修改。
  4. 送出 Pull Request,並在描述裡引用這個 Issue 編號。

huggingface/diffusers 的其他 Issue

查看 huggingface/diffusers 的全部 Issue

相似的 Issue

更多 Python Issue

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。