enable_model_cpu_offload(): first post-load generation thrashes the Windows pagefile (~6x step time), silently recovers

Open
  • Is something wrong?
#4,488 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
Half a day
Newbie friendliness
35/100
Issue type
Documentation
Clarity
Mostly clear
Activity status
Active
Tech stack
python, pytorch
Domain
documentation, performance

Research direction

Start from enable_model_cpu_offload() and the cpu_offload documentation the issue asks to extend. The report is a Windows pagefile slowdown on the first generation after load, so the first step is to reproduce the timing pattern or confirm the explanation before writing anything. Done means a docs note describing the first-run slowdown on Windows hosts whose weights approach system RAM, reviewed by a maintainer who has confirmed the wording.

Written by the indexing model from the issue text.

Description

enable_model_cpu_offload(): first post-load generation thrashes the

Windows pagefile (~6x step time), silently recovers on the next run

Environment: Windows 11 26300, 32 GB RAM (31.8 usable), pagefile
system-managed (grew to ~55 GB cap, ~20 GB in use at observation), RX 7900
XTX 24 GB, torch 2.12.0+rocm7.14.1, accelerate 1.15.0, diffusers
QwenImage21Pipeline with enable_model_cpu_offload() (weights total
30.86 GB host-side: 7B DiT + 8B text encoder + VAE, bf16).

Measured (fixed seed/protocol, median per run):

run (back-to-back, same process config) steady s/step @1024²
cold (first load; OS file cache hot) 3.6–3.7
warmup run immediately after 24–26 (6-7x slower)
measure run right after that 3.66 (fully recovered)

During the slow run the pagefile grew by ~20 GB. Interpretation: with
weights ≈ system RAM, the first post-load generation forces Windows to
commit/grow the pagefile while cpu_offload streams module weights;
the growth cost lands on that run's steps. Steady state has no per-step
paging (DiT stays resident in VRAM; only text-encoder transfers per
prompt). It self-recovers, so it is not a correctness bug — but for other
~32 GB-RAM Windows users the symptom ("first image after loading is
~6x slower, then everything is fine") is alarming and underdocumented.

Ask:

  1. A note in the cpu_offload docs: on Windows hosts where total weights
    approach system RAM, expect one sacrificial slow generation after the
    first load (pagefile commit), recovering on subsequent runs.
  2. (Stretch) Consider an explicit torch.cuda.empty_cache()-style hint
    or a warning when total_weights_bytes > 0.8 * total_ram that the
    first run may page-thrash, so benchmarks discard it.

Raw protocol data: 40-step runs, median-of-3 per run class
(bench evidence).

Dominant language
Python
Stars
9.9k
Forks
1.6k
Avg merge
2d 13h
Merged PRs (30d)
13

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from huggingface/accelerate

All issues in huggingface/accelerate

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.