[Bug] Unmerging accumulated LoRA adapters drops earlier adapters from inference

Open Beginner friendly
  • Is something wrong?
#2,096 4 comments 1 reaction 0 assignees View on GitHub

Maintainers usually reply within 2 days

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
76/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
python, pytorch
Domain
machine-learning

Research direction

Start in fastvideo/layers/lora/linear.py by reading set_lora_weights(), the unmerged forward, and unmerge_lora_weights(); reproduce the supplied BF16 example on the specified commit with FASTVIDEO_LORA_SIDE_BRANCH=0. Add a regression test checking forward outputs before unmerge, after unmerge, and after remerge; done means all three match the expected accumulated-adapter output of 34.

Written by the indexing model from the issue text.

Description

platform scope: inference scope: training
Describe the bug

On a BF16 layer with two accumulated adapters, unmerge_lora_weights() drops the first adapter’s contribution from the next forward pass. Calling merge_lora_weights() again still produces the output for the second adapter alone.

For a base weight of 10, adapters X (+2) and Y (+5), and input 2, the merged output is 34. After unmerge it is 30, and after remerge it is still 30. Expected 34 in all three modes. The scalar values are exactly representable in BF16.

Reproduction

Run against ec9145ab3d86b2255b6fa3bbbb2845859c72cafd with FASTVIDEO_LORA_SIDE_BRANCH=0.

from unittest.mock import patch

import torch
from fastvideo.layers.linear import ReplicatedLinear
from fastvideo.layers.lora import linear

d = torch.bfloat16
with torch.no_grad(), torch._dynamo.config.patch(disable=True), patch.object(
    linear, "get_local_torch_device", return_value=torch.device("cpu")
):
    base = ReplicatedLinear(1, 1, bias=False, params_dtype=d)
    base.weight.fill_(10)
    layer = linear.BaseLayerWithLoRA(base)
    x = torch.tensor([[2]], dtype=d)
    a = torch.ones(1, 1, dtype=d)

    layer.set_lora_weights(a, torch.tensor([[2]], dtype=d))       # X adds 2
    layer.set_lora_weights(a, torch.tensor([[5]], dtype=d), accumulate=True)  # Y adds 5
    print(layer(x)[0].item())   # 34
    layer.unmerge_lora_weights()
    print(layer(x)[0].item())   # 30, expected 34
    layer.merge_lora_weights()
    print(layer(x)[0].item())   # 30, expected 34

The layer overwrites its current factor pair on every adapter update, including accumulation. The merged weight contains X and Y, but the current factor pair belongs to Y. Unmerge restores the base weight, and the unmerged forward applies only Y. A regression test should check both forward outputs across this round trip.

Environment

FastVideo main ec9145ab3d86b2255b6fa3bbbb2845859c72cafd, Python 3.12.13, PyTorch 2.12.0, macOS arm64. Reproduced on CPU with one unquantized BF16 ReplicatedLinear. This tracks the general adapter lifecycle behavior observed while working on the NVFP4 restore/repack change in #2076.

Dominant language
Python
Stars
4.6k
Forks
486
Avg merge
3d 15h
Merged PRs (30d)
74

Getting set up

  • No Dockerfile or Docker Compose file
  • Has a pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from hao-ai-lab/FastVideo

All issues in hao-ai-lab/FastVideo

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.