Dynamic VRAM applies LoRAs in the weight's dtype: on bf16 models, part of the LoRA rounds away

Open
  • Is something wrong?
#17,690 1 comment 2 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
77/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
python, pytorch
Domain
backend, machine-learning, performance

Research direction

Read LowVramPatch.__call__ in comfy/model_patcher.py and the dynamic path in comfy/ops.py (post_cast), then compare with patch_weight_to_device and lora_compute_dtype() as described in the issue. Check how the sum's intermediate dtype and result cast affect LoRA deltas on bf16 weights. Done when dynamic and static VRAM paths retain comparable LoRA changes without changing the output weight dtype.

Written by the indexing model from the issue text.

Description

What happens

With dynamic VRAM on (the default), a LoRA on a bf16-compute model is applied weaker than with --disable-dynamic-vram. The same workflow, seed and LoRA give visibly different pictures in the two modes.

Cause

LowVramPatch.__call__ (comfy/model_patcher.py) computes the patch with intermediate_dtype=weight.dtype:

return comfy.lora.calculate_weight(patches, weight, self.key, intermediate_dtype=weight.dtype)

In the dynamic path (comfy/ops.py, post_cast), the weight has already been cast or dequantized to the compute dtype, so on a bf16 model W + dW is summed in bf16. A LoRA's per-weight change is usually far below bf16's step at the weight's magnitude, so most of the sums round back to W. The static path uses lora_compute_dtype() (fp16 or fp32), which keeps the delta.

Measured

MiniMax H3 Image (minimax_h3_image_pruned_int8_convrot.safetensors, supported_inference_dtypes = [bfloat16, float32]), a rank-32 LoRA on 200 Linears, |dW|/|W| 0.06–0.5% per layer:

Sum done in LoRA kept (projection of the applied delta on dW) Weights left unchanged
bf16 0.85 overall, 0.44–0.99 per layer 36–86%
fp16 0.999 7–39%

Renders, same seed, 1536×1536, 6 steps (mean abs pixel difference, 0–255; two renders of the same setup differ by about 4):

  • dynamic VRAM on vs --disable-dynamic-vram: 15.3
  • a reference implementation applying the exact LoRA vs --disable-dynamic-vram: 5.7
  • the same reference with only the W + dW sum moved to bf16 vs dynamic VRAM on: 3.5

Minimal reproduction of the rounding (torch only)

import torch
torch.manual_seed(0)
W = torch.randn(4096, 4096) * 0.02
up, down = torch.randn(4096, 32) * 0.002, torch.randn(32, 4096) * 0.002
dW = up @ down                                   # |dW|/|W| ~ 0.1%
for dt in (torch.float32, torch.float16, torch.bfloat16):
    Wl = W.to(dt)
    got = (Wl + dW.to(dt)).float() - Wl.float()
    print(dt, "kept %.3f" % ((got * dW).sum() / (dW * dW).sum()),
          "unchanged %.0f%%" % (100 * (got == 0).float().mean()))
# float32 kept 1.000 unchanged 0%
# float16 kept 1.000 unchanged 20%
# bfloat16 kept 0.573 unchanged 75%

Suggested fix

Do the sum in comfy.model_management.lora_compute_dtype(device) (as patch_weight_to_device does) and cast the result back, instead of weight.dtype.

Version

ComfyUI 0.45.0 (4d64a8b, 11 Nov 2026), Windows, RTX 5090, PyTorch CUDA build. Reproduced with --disable-all-custom-nodes.

Dominant language
Python
Stars
146k
Forks
17.4k
Avg merge
19h 44m
Merged PRs (30d)
178

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from Comfy-Org/ComfyUI

All issues in Comfy-Org/ComfyUI

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.