Dynamic VRAM (comfy-aimdo 0.6.0) on Windows/AMD: ~2.4 GB of freshly loaded MOSSImage weights occasionally land in shared GPU memory despite ~6 GB free VRAM, sampling becomes ~5x slower (random, ~12% of runs)

Aperta
  • Qualcosa non va?
#17,656 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
35/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
python, pytorch
Ambito
backend, performance

Direzione di ricerca

The report points to ComfyUI's dynamic VRAM loading path and comfy-aimdo's WDDM handling, but names no source files or tests. Start by tracing where ModelPatcherDynamic loads MOSSImage weights and how aimdo reports resident versus shared memory; compare the supplied slow and normal run measurements. Done means identifying why about 2.4 GB is intermittently placed in shared memory and documenting a reproducible diagnosis or fix.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Environment

  • ComfyUI 0.43.0, Windows 10 Pro (10.0.19045), Python 3.12.9
  • PyTorch 2.14.0+rocm10.1.0, ROCm version (7, 16), AMD Radeon RX 6900 XT (gfx1030), 16368 MB VRAM, 32 GB system RAM, AMD driver 32.0.21045.16010
  • comfy-aimdo 0.6.0 (latest on PyPI), aimdo log: shmem-detect.c ... WDDM adapter match: AMD Radeon RX 6900 XT, cuda-detour.c: installing 6 hooks
  • Launch args: --listen (IP) --port (port) --disable-smart-memory --use-quad-cross-attention --fp16-unet --disable-pinned-memory --vram-headroom 2.5
  • Workflow: MOSS Image (moss_anime_turbo_sft_int8_convrot, ~5.2 GB, ModelPatcherDynamic) + Qwen3.5-9B text encoder (17 GB staged) + a separate llama-server process for prompt enhancement (sleeps between runs). Models are unloaded after every prompt.

Problem

Normally a run takes ~135-140 s (sampling 83 s). In about 1 of 8 runs the sampling step takes 430-480 s while all other phases are unchanged. Reproduced twice with full measurements today (11:35 and 13:15), and several more times earlier.

What I measured (GPU memory counters once per second, Windows "GPU Adapter Memory" dedicated / shared, plus per process)

Right after "Requested to load MOSSImage" (text encoder gets unloaded, MOSSImage prepared for dynamic VRAM):

normal runs slow runs (11:35 / 13:15)
dedicated VRAM 1-3 s after load ~12.2 GB 9.95 GB / 10.0 GB
shared GPU memory ~0.46 GB 2.69 GB / 2.61 GB
dedicated + shared ~12.6 GB ~12.6 GB (same)
get_total_vram_usage() (aimdo) 12.45-12.69 GB 11.56 GB
ModelPatcherDynamic loaded_mb of 5163 3137-3361 2241
  • The same amount is allocated, but ~2.4 GB ends up in shared (system RAM backed) memory instead of dedicated VRAM, although ~6 GB of VRAM was free at that moment. Dedicated stays pinned near 10 GB in both slow runs, even though the state before the load differed (11.3 GB resp. 15.4 GB).
  • It is decided within about 1-2 s after the text encoder is unloaded and MOSSImage is loaded. It never changes during sampling afterwards, which then runs ~5x slower (weights are streamed from system memory every step).
  • Torch allocator state before the load is clean in slow and normal runs (reserved 2 MB, no alloc retries). Other GPU processes unchanged (browser, Discord constant). RAM free at load was comparable (7-12 GB) in slow and normal runs.
  • --vram-headroom 3.5 (instead of 2.5) made no difference (slow run happened anyway).
  • Same pattern in the separate llama-server process (9.5 GB GGUF loaded with -ngl auto --fit on): in some runs ~2.4 GB lands in shared memory instead of VRAM (LLM phase 30 s -> 55-90 s), so this may be a driver/WDDM-level demotion rather than something specific to MOSS.
  • With --disable-dynamic-vram the machine ran out of virtual memory (Windows error 1455, whole system froze), so that is not a usable workaround here.

Question

Is there a known race / budget issue on WDDM (e.g. VRAM from the just-unloaded text encoder not yet returned when the new vbar maps its pages, or a WDDM per-process budget decision) that could produce this? Is there a way to log what aimdo decides for the vbar at load time (resident vs. host pages)? I can provide more logs, the per-second GPU traces and per-run summaries.

lauf-1135-summary.json
lauf-1315-log-slice.txt
lauf-1315-summary.json
lauf-1135-log-slice.txt

Lingua principale
Python
Stelle
146k
Fork
17.4k
Merge medio
19h 44m
PR unite (30g)
178

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di Comfy-Org/ComfyUI

Tutte le issue di Comfy-Org/ComfyUI

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.