Possible training qwen-image lora quality regression on Blackwell GPUs (e.g., RTX 5090)

Possible training qwen-image lora quality regression on Blackwell GPUs (e.g., RTX 5090)

Hello, I really like this project!
I have used it to train many LoRA models, and it has been great.

Recently, I purchased an RTX 5090 GPU.
To support it, I installed:

• pytorch==2.8.0
• cuda==12.9
• bitsandbytes==0.47 (this version added support for CUDA 12.9)

When i test train a qwen-image lora . The program runs without errors.
However, under the same training parameters but on different hardware(autodl) configurations, the training quality on the Blackwell 5090 seems worse.

I am not sure how to address this issue. Thanks for your help!

---

Attachments

• Sample comparison images (see below)
• pip dependency list
• train.bat and related files

---

this is my pip list

(venv) PS D:\AI\musubi-tuner> pip freeze absl-py==2.3.1 accelerate==1.6.0 av==14.0.1 bitsandbytes==0.47.0 certifi==2025.8.3 charset-normalizer==3.4.3 colorama==0.4.6 diffusers==0.32.1 easydict==1.13 einops==0.7.0 filelock==3.18.0 fsspec==2025.7.0 ftfy==6.3.1 grpcio==1.74.0 huggingface-hub==0.34.3 idna==3.10 importlib_metadata==8.7.0 Jinja2==3.1.6 Markdown==3.8.2 MarkupSafe==3.0.2 mpmath==1.3.0 -e git+https://github.com/kohya-ss/musubi-tuner.git@b3fe46d1027266a7996f46bc55f5de8ed6bf1203#egg=musubi_tuner networkx==3.4.2 numpy==2.2.6 opencv-python==4.10.0.84 packaging==25.0 pillow==11.3.0 protobuf==6.31.1 psutil==7.0.0 PyYAML==6.0.2 regex==2025.7.34 requests==2.32.4 safetensors==0.4.5 sentencepiece==0.2.0 sympy==1.14.0 tensorboard==2.20.0 tensorboard-data-server==0.7.2 tokenizers==0.21.4 toml==0.10.2 torch @ file:///E:/%E8%BF%85%E9%9B%B7%E4%B8%8B%E8%BD%BD/torch-2.8.0%2Bcu129-cp310-cp310-win_amd64.whl#sha256=e4adcc3d44089d4a696e6a2ca4233d1ddad7614adc1d48ec8a8cfb95ba235ea1 torchvision==0.23.0 tqdm==4.67.1 transformers==4.54.1 typing_extensions==4.14.1 urllib3==2.5.0 voluptuous==0.15.2 wcwidth==0.2.13 Werkzeug==3.1.3 zipp==3.23.0

this is my musubi-tuner version

(venv) PS D:\AI\musubi-tuner> git rev-parse HEAD b3fe46d1027266a7996f46bc55f5de8ed6bf1203 (venv) PS D:\AI\musubi-tuner> git describe --tags --always v0.2.7-17-gb3fe46d

train.bat

accelerate launch --num_cpu_threads_per_process 1 --mixed_precision bf16 src/musubi_tuner/qwen_image_train_network.py ^ --dit D:\AI\Qwen-model\qwen_image_bf16.safetensors ^ --vae D:\AI\Qwen-model\vae.safetensors ^ --text_encoder D:\AI\Qwen-model\qwen_2.5_vl_7b.safetensors ^ --dataset_config D:\AI\train\QWEN_IMAGE\sugo-v001-qw\dataset.toml ^ --sdpa --mixed_precision bf16 ^ --timestep_sampling shift ^ --weighting_scheme none --discrete_flow_shift 3.0 ^ --optimizer_type adamw8bit --learning_rate 1e-4 ^ --gradient_checkpointing ^ --max_data_loader_n_workers 2 --persistent_data_loader_workers ^ --network_module musubi_tuner.networks.lora_qwen_image ^ --network_dim 32 ^ --network_alpha 1 ^ --max_train_epochs 16 --save_every_n_epochs 2 --seed 42 ^ --output_dir D:\AI\train\QWEN_IMAGE\sugo-v001-qw\output ^ --blocks_to_swap 16 ^ --sample_prompts D:\AI\train\QWEN_IMAGE\sugo-v001-qw\sample_prompts.txt ^ --output_name sugo-ip-qwen ^ --sample_every_n_epochs 1 --sample_at_first ^ --logging_dir=logs ^ --fp8_base ^ --fp8_scaled ^

loss image

and this is test image