Dynamic VRAM: second offload stream's cast buffer first grows after block 0's activations, "VRAM grow failed: 424159232 bytes" on 15 s FastH3 I2V/R2V

Open
  • Is something wrong?
#17,702 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
52/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
python, pytorch
Domain
backend, machine-learning

Research direction

Start in comfy/model_management.py at get_aimdo_cast_buffer and trace how per-stream buffers are keyed and grown; then follow the cast path in comfy/ops.py and the prefetch call in comfy/model_prefetch.py. Reproduce with the reported 15-second FastH3 I2V workflow on a 12 GB GPU, and compare buffer reservation or borrowing approaches. Done when that workflow runs with the default stream count.

Written by the indexing model from the issue text.

Description

With Dynamic VRAM, each async offload stream gets its own aimdo cast buffer (get_aimdo_cast_buffer in comfy/model_management.py, keyed on STREAM_AIMDO_CAST_BUFFERS[offload_stream]), grown on first use to the largest casted weight (424 MB for an H3 int8 block). Block 0 uses stream A. Its activations then leave very little VRAM, and block 1 is the first use of stream B, whose buffer must grow by 424 MB:

  File "comfy/ldm/minimax/model.py", line 658, in _run_blocks
    comfy.model_prefetch.prefetch_queue_pop(prefetch_queue, device, block, malloc_scope="block")
  File "comfy/model_prefetch.py", line 121, in pin_modules
    offload_stream, fully_faulted = comfy.ops.cast_modules_with_vbar(comfy_modules, None, device, None, True, return_faulted=True)
  File "comfy/ops.py", line 252, in cast_modules_with_vbar
    xfer_dest = get_cast_buffer(dest_size)
  File "comfy/ops.py", line 179, in get_cast_buffer
    buffer = comfy_aimdo.torch.aimdo_to_tensor(cast_buffer.get(buffer_size, cast_buffer_offset), device)
  File "comfy_aimdo/vram_buffer.py", line 49, in get
    raise RuntimeError(f"VRAM grow failed: {required_size} bytes")
RuntimeError: VRAM grow failed: 424159232 bytes

A per-block memory log (torch.cuda.mem_get_info() at each block start) shows the drop:

h3_memlog block 0 start: torch_alloc=410MB torch_reserved=448MB cuda_free=7052MB
h3_memlog block 1 start: torch_alloc=541MB torch_reserved=576MB cuda_free=148MB

Environment: ComfyUI v0.45.0 (code also checked unchanged on master b2f6707443), comfy-aimdo 0.6.0, comfy-kitchen 0.2.41, PyTorch 2.15.0+cu132, RTX 3060 12 GB, Linux, Dynamic VRAM on (default). Workflow: the official video_fastvideo_fasth3_t2v / i2v / video_fasth3_ominiref_r2v templates with length 362 (15 s) at 1344x768, about 109k video tokens. It fails on image-to-video (first-frame conditioning) and reference-to-video with FastH3 at 15 s, four times on 2026-11-11. Text-to-video at the same length passed, apparently because stream B's buffer had already grown earlier in the run.

Workaround. --async-offload 1 (one stream, one buffer) fixes it at about 10-13 % slower steps.

What I tried. Growing both streams' buffers to the largest weight size before the forward crashed with cudaErrorInvalidValue from cuMemFreeAsync twice, so that is not a safe fix as written.

Needed. Reserve every stream's cast buffer at the same time (before activations exist), or let a stream borrow the other's buffer when growing fails. Done when 15 s FastH3 I2V runs on 12 GB with the default stream count.

Filed by Claude Opus 5.5 in Claude Code.

Dominant language
Python
Stars
146k
Forks
17.4k
Avg merge
19h 44m
Merged PRs (30d)
178

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from Comfy-Org/ComfyUI

All issues in Comfy-Org/ComfyUI

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.