Flux Control and HunyuanVideo LoRA converters swap the scale/shift halves of `lora_A` in `final_layer`

Offen
#15,305 2 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen

Maintainer antworten meist innerhalb von 1 Tag

@MaiAditya arbeitet bereits daran.

Seit 16.11.2026.

  • #15307 von @MaiAditya — offen

Bewertung

Schwierigkeit
2/5
Geschätzter Aufwand
1-3 Stunden
Anfängerfreundlichkeit
35/100
Issue-Typ
Bug
Klarheit
Klar beschrieben
Aktivitätsstatus
Veraltet
Tech-Stack
python, pytorch

Rechercherichtung

Read loaders/lora_conversion_utils.py and the existing test_lora_bfl_final_layer_modulation test. The issue identifies both affected converters and proposes one test per converter, checking that only lora_B is swapped; run those tests to confirm the converted delta matches the expected scale/shift order. A pull request is already open for this issue.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Beschreibung

Describe the bug

final_layer.adaLN_modulation.1 in the original checkpoints orders its output (shift, scale), and diffusers' norm_out.linear orders it (scale, shift), so the converters swap the two halves. For a LoRA only lora_B should be swapped: its rows are the layer's outputs. lora_A is (rank, in), so swapping its halves reorders the rank dimension and the converted delta is wrong.

Two converters in loaders/lora_conversion_utils.py swap both:

  • _convert_bfl_flux_control_lora_to_diffusers: swap_scale_shift is applied for every lora_key, including lora_A (weight and bias).
  • _convert_hunyuan_video_lora_to_diffusers: remap_norm_scale_shift_ swaps every final_layer.adaLN_modulation.1 key, lora_A included.

The kohya Flux converter already swaps only lora_B. This is the same class of bug #15276 fixed for Flux2.

Reproduction
import torch
from diffusers.loaders.lora_conversion_utils import _convert_bfl_flux_control_lora_to_diffusers

RANK, INNER = 4, 3072
torch.manual_seed(0)

class FillingDict(dict):
    # dummy tensors for the keys the converter pops that this repro doesn't care about
    def pop(self, key, *default):
        if key in self:
            return super().pop(key)
        if default:
            return default[0]
        return torch.zeros(21504, RANK) if ".lora_B." in key else torch.zeros(RANK, 8)

lora_A = torch.randn(RANK, INNER)
lora_B = torch.randn(2 * INNER, RANK)  # rows: (shift, scale)
sd = FillingDict({
    "final_layer.adaLN_modulation.1.lora_A.weight": lora_A.clone(),
    "final_layer.adaLN_modulation.1.lora_B.weight": lora_B.clone(),
    "double_blocks.0.img_attn.norm.query_norm.scale": torch.ones(1),
})
out = _convert_bfl_flux_control_lora_to_diffusers(sd)
A = out["transformer.norm_out.linear.lora_A.weight"]
B = out["transformer.norm_out.linear.lora_B.weight"]
shift, scale = (lora_B @ lora_A).chunk(2, dim=0)
expected = torch.cat([scale, shift])
print(torch.allclose(B @ A, expected, atol=1e-4), (B @ A - expected).abs().max().item())
print(torch.allclose(B @ lora_A, expected, atol=1e-4))  # lora_A left as is
Logs
False 39.052650451660156
True

The HunyuanVideo converter gives the same result (max error 17.05 in the equivalent check).

System Info

diffusers main @ 43e3002, torch 2.15.0, Python 3.12, macOS (CPU).

Who can help?

@sayakpaul

I have a fix with tests for both converters (swap only lora_B; one test per converter in the style of test_lora_bfl_final_layer_modulation, both fail on main) and can open a PR if you'd like. It may change the expected slices of the slow Flux Control LoRA integration tests, which I can't regenerate without a GPU.

Vorherrschende Sprache
Python
Sterne
34.9k
Forks
7.5k
Ø Merge
3 T. 16 Std.
Gemergte PRs (30 T.)
65

Entwicklungsumgebung

Erste Schritte

  1. Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
  2. Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
  3. Forken Sie das Repository und arbeiten Sie in einem Branch.
  4. Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.

Mehr aus huggingface/diffusers

Alle Issues in huggingface/diffusers

Ähnliche Issues

Weitere Issues zu Python

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.