基于Qwen-Image-2.1进行Lora微调，效果很快变差，而同样的任务，基于FLUX2-Klein-base-9B微调效果在变好而没有变差，是什么问题？该如何解决？

Aberta
  • Algo errado?
#1,756 4 comentários 0 reações 0 responsáveis Ver no GitHub

Mantenedores costumam responder em até 1 dia

Ninguém assumiu esta issue ainda.

Avaliação

Dificuldade
5/5
Tempo estimado
Mais de uma semana
Facilidade para iniciantes
15/100
Tipo de issue
Bug
Clareza
Precisa de esclarecimento
Status de atividade
Ativa
Stack de tecnologia
python
Domínio
machine-learning

Direção de pesquisa

The FLUX2 LoRA run is launched from examples/flux2/model_training/train.py, and the reporter's Qwen-Image-2.1 LoRA run is the one that degrades. Start by comparing the Qwen training entry point and its LoRA target modules against the FLUX2 script to find where the two diverge. Done would mean an explanation of the quality drop and a reproducible fix, which likely needs maintainer input on whether Qwen-Image-2.1 supports LoRA fine-tuning.

Escrita pelo modelo de indexação a partir do texto da issue.

Descrição

设置

  • 代码 commit=39da7b9dbebd06af848b324a5b3a976c6d8e7839
  • 训练数据：3万条已经验证过的鞋靴穿搭多参考图数据
  • 测试数据： 1条样本
  • env 环境
点击展开/折叠查看依赖库及版本 ``` absl-py==2.4.0 accelerate==1.10.1 addict==2.4.0 aether==0.1.3.post1 aiobotocore==2.17.0 aiohappyeyeballs==2.6.1 aiohttp==3.13.5 aiohttp-cors==0.8.1 aioitertools==0.13.0 aiosignal==1.4.0 annotated-doc==0.0.4 annotated-types==0.7.0 anthropic==0.102.0 antlr4-python3-runtime==4.9.3 anyio==4.13.0 apache-tvm-ffi==0.1.8.post2 apex @ file:///opt/apex archspec @ file:///croot/archspec_1709217642129/work astor==0.8.1 astunparse==1.6.3 attrdict==2.0.1 attrs==26.1.0 audioread==3.1.0 autoawq==0.2.7.post3 autocommand==2.2.2 av==13.1.0 backports.tarfile==1.2.0 badam==1.2.2 bitarray==3.8.1 bitsandbytes==0.49.2 bitstring==4.4.0 blake3==1.0.8 blessed==1.39.0 blinker==1.9.0 boltons @ file:///croot/boltons_1737061692168/work boto3==1.35.54 botocore==1.35.93 braceexpand==0.1.7 bracex==2.6 brotli==1.2.0 cachetools==7.1.2 cbor2==6.1.1 certifi==2026.4.22 cffi==2.0.0 chardet==6.0.0.post1 charset-normalizer==3.4.7 click==8.4.0 cloudpickle==3.1.2 cmake==4.3.2 codetiming==1.4.0 colorama==0.4.6 colorful==0.5.8 compressed-tensors==0.13.0 conda @ file:///croot/conda_1738168370816/work conda-anaconda-telemetry @ file:///croot/conda-anaconda-telemetry_1736524596117/work conda-anaconda-tos @ file:///croot/conda-anaconda-tos_1739298990519/work conda-content-trust @ file:///croot/conda-content-trust_1714483159009/work conda-libmamba-solver @ file:///croot/conda-libmamba-solver_1737733694612/work/src conda-package-handling @ file:///croot/conda-package-handling_1731369017509/work conda_package_streaming @ file:///croot/conda-package-streaming_1731366181659/work confluent-kafka==2.13.2 contourpy==1.3.3 coverage==7.14.0 cpm-kernels==1.0.11 crc32c==2.8 cryptography==48.0.0 cuda-bindings==12.9.4 cuda-pathfinder==1.5.4 cuda-python==12.9.4 cupy-cuda12x==14.0.1 cycler==0.12.1 dacite==1.8.1 daft==0.7.13 DataProperty==1.1.1 datasets==4.8.4 decorator==5.3.1 decord==0.6.0 deep_ep==1.2.1 deepspeed==0.16.9 defusedxml==0.7.1 Deprecated==1.3.1 depyf==0.20.0 # Editable Git install with no remote (diffsynth==2.1.9) -e /home/ImageGen/DiffSynth-Studio-1013 diffusers==0.35.1 dill==0.4.1 diskcache==5.6.3 distlib==0.4.0 distro @ file:///croot/distro_1714488253808/work dnspython==2.8.0 docstring_parser==0.16 eametric==1.0.dev0 easydict==1.13 ebmlite==3.4.1 einops==0.8.0 elastic-transport==8.17.1 elasticsearch==8.15.1 email-validator==2.3.0 et_xmlfile==2.0.0 evaluate==0.4.6 fairscale==0.4.13 faiss-cpu==1.13.2 fastapi==0.133.1 fastapi-cli==0.0.24 fastapi-cloud-cli==0.17.1 fastar==0.11.0 fbgemm_gpu==1.5.0+cu128 filelock==3.29.0 filetype==1.2.0 fire==0.7.1 fla-core==0.5.0 FlagEmbedding==1.4.0 flash-linear-attention==0.5.0 flash_attn==2.7.4.post1 flashinfer-python==0.6.6 Flask==3.1.0 flask-cors==5.0.1 fonttools==4.63.0 frozendict @ file:///croot/frozendict_1713194832637/work frozenlist==1.8.0 fsspec==2026.1.0 ftfy==6.3.1 gguf==0.19.0 gitdb==4.0.12 GitPython==3.1.43 google-api-core==2.29.0 google-auth==2.53.0 googleapis-common-protos==1.73.0 gpustat==1.1.1 GPUtil==1.4.0 gradio==6.14.0 gradio_client==2.5.0 groovy==0.1.2 grpcio==1.80.0 grpcio-tools==1.80.0 h11==0.16.0 hf-gradio==0.4.1 hf-xet==1.5.0 hjson==3.1.0 httpcore==1.0.9 httptools==0.7.1 httpx==0.28.1 httpx-sse==0.4.3 huggingface_hub==1.15.0 hydra-core==1.3.2 idna==3.15 ijson==3.5.0 ImageIO==2.38.1 imageio-ffmpeg==0.6.0 importlib_metadata==8.7.1 iniconfig==2.3.0 interegular==0.3.3 iopath==0.1.10 itsdangerous==2.2.0 jaraco.context==6.1.0 jaraco.functools==4.4.0 jaraco.text==4.0.0 jieba==0.42.1 Jinja2==3.1.6 jiter==0.14.0 jmespath==1.1.0 joblib==1.5.3 jsonlines==4.0.0 jsonpatch @ file:///croot/jsonpatch_1714483231291/work jsonpointer==2.1 jsonschema==4.26.0 jsonschema-specifications==2025.9.1 kafka-python==2.0.2 kiwisolver==1.5.0 lance-namespace==0.7.6 lance-namespace-urllib3-client==0.7.6 lark==1.2.2 latex2mathml==3.81.0 lazy-loader==0.5 libmambapy @ file:///croot/mamba-split_1734469461757/work/libmambapy librosa==0.11.0 lightning-utilities==0.15.3 llguidance==1.3.0 llvmlite==0.44.0 lm-format-enforcer==0.11.3 lm_eval @ file:///opt/lm-evaluation-harness loguru==0.7.3 lxml==6.1.0 Markdown==3.10.2 markdown-it-py==4.2.0 MarkupSafe==3.0.3 mathruler==0.1.0 matplotlib==3.10.9 -e git+https://github.com/ISEEKYAN/mbridge.git@641a5a01de71080b2200d10e369090e40c9a351c#egg=mbridge mbstrdecoder==1.1.5 mcp==1.27.1 mdtex2html==1.3.0 mdurl==0.1.2 -e git+https://github.com/NVIDIA-NeMo/Megatron-Bridge.git@7bea48e09631a3fd97b7e08b91899aec949a4be5#egg=megatron_bridge # Editable Git install with no remote (megatron-core==0.16.0) -e /opt/Megatron-LM megatron-energon==6.0.1 menuinst @ file:///croot/menuinst_1738943416351/work mistral_common==1.11.2 ml_dtypes==0.5.4 mmengine==0.10.7 mmengine-lite==0.10.7 mmh3==5.0.1 model-hosting-container-standards==0.1.15 modelscope==1.40.2 modelscope-hub==0.4.6 more-itertools==11.0.2 mpi4py==4.1.2 mpmath==1.3.0 msgpack==1.1.2 msgspec==0.21.1 multi-storage-client==0.48.0 multidict==6.7.1 multiprocess==0.70.19 mypy_extensions==1.1.0 natsort==8.4.0 networkx==3.6.1 ninja==1.13.0 nltk==3.9.1 numba==0.61.2 numexpr==2.14.1 numpy==2.2.6 nvidia-cublas-cu12==12.8.4.1 nvidia-cuda-cupti-cu12==12.8.90 nvidia-cuda-nvrtc-cu12==12.8.93 nvidia-cuda-runtime-cu12==12.8.90 nvidia-cudnn-cu12==9.10.2.21 nvidia-cudnn-frontend==1.18.0 nvidia-cufft-cu12==11.3.3.83 nvidia-cufile-cu12==1.13.1.3 nvidia-curand-cu12==10.3.9.90 nvidia-cusolver-cu12==11.7.3.90 nvidia-cusparse-cu12==12.5.8.93 nvidia-cusparselt-cu12==0.7.1 nvidia-cutlass-dsl==4.4.2 nvidia-cutlass-dsl-libs-base==4.4.2 nvidia-ml-py==12.575.51 nvidia-modelopt==0.44.0 nvidia-nccl-cu12==2.27.5 nvidia-nvjitlink-cu12==12.8.93 nvidia-nvshmem-cu12==3.4.5 nvidia-nvtx-cu12==12.8.90 nvidia-resiliency-ext==0.6.0 nvtx==0.2.15 omegaconf==2.3.0 onnx==1.19.0 onnx-ir==0.2.1 onnxscript==0.7.0 openai==2.24.0 openai-harmony==0.0.4 opencensus==0.11.4 opencensus-context==0.1.3 opencv-python==4.10.0.84 opencv-python-headless==4.13.0.92 openpyxl==3.1.5 opentelemetry-api==1.41.1 opentelemetry-exporter-otlp==1.41.1 opentelemetry-exporter-otlp-proto-common==1.41.1 opentelemetry-exporter-otlp-proto-grpc==1.41.1 opentelemetry-exporter-otlp-proto-http==1.41.1 opentelemetry-exporter-prometheus==0.62b1 opentelemetry-proto==1.41.1 opentelemetry-sdk==1.41.1 opentelemetry-semantic-conventions==0.62b1 opentelemetry-semantic-conventions-ai==0.5.1 optimum==2.1.0 orjson==3.11.9 outlines_core==0.2.11 packaging==25.0 pandas==2.3.3 parameterized==0.9.0 partial-json-parser==0.2.1.1.post7 pathvalidate==3.3.1 peft==0.18.1 pillow==12.2.0 platformdirs==4.9.6 pluggy==1.6.0 pooch==1.9.0 portalocker==3.2.0 prettytable==3.17.0 prometheus-fastapi-instrumentator==7.1.0 prometheus_client==0.22.1 propcache==0.5.2 proto-plus==1.27.1 protobuf==5.29.6 psutil==7.2.2 PuLP==2.9.0 py-cpuinfo==9.0.0 py-spy==0.4.2 pyarrow==23.0.1 pyasn1==0.6.3 pyasn1_modules==0.4.2 pybase64==1.4.2 pybind11==3.0.4 pycosat @ file:///croot/pycosat_1736868416091/work pycountry==26.2.16 pycparser==3.0 pydantic==2.13.4 pydantic-extra-types==2.11.1 pydantic-settings==2.14.1 pydantic_core==2.46.4 pydub==0.25.1 Pygments==2.20.0 PyJWT==2.12.1 pylance==4.0.0 pylatexenc==2.10 pynvml==12.0.0 pyparsing==3.3.2 pyre-extensions==0.0.32 pytablewriter==1.2.1 pytest==8.4.1 pytest-asyncio==1.3.0 pytest-cov==6.2.1 pytest-html==4.1.1 pytest-metadata==3.1.1 python-dateutil==2.9.0.post0 python-discovery==1.3.1 python-dotenv==1.2.2 python-json-logger==4.1.0 python-multipart==0.0.29 pytz==2026.2 pyvers==0.1.0 PyYAML==6.0.3 pyzmq==27.1.0 quack-kernels==0.3.7 qwen-vl-utils==0.0.14 ray==2.51.1 referencing==0.37.0 regex==2026.5.9 requests==2.34.2 rich==15.0.0 rich-toolkit==0.19.9 rignore==0.7.6 rouge-chinese==1.0.3 rouge_score==0.1.2 rpds-py==0.30.0 ruff==0.14.6 s3fs==2026.1.0 s3transfer==0.10.4 sacrebleu==2.6.0 safehttpx==0.1.7 safetensors==0.7.0 scikit-build==0.19.0 scikit-learn==1.8.0 scipy==1.17.1 seaborn==0.13.2 semantic-version==2.10.0 sentence-transformers==5.2.3 sentencepiece==0.2.1 sentry-sdk==2.60.0 setproctitle==1.3.7 setuptools==80.10.2 shellingham==1.5.4 shortuuid==1.0.13 shtab==1.7.1 six==1.17.0 smart_open==7.6.1 smmap==5.0.3 sniffio==1.3.1 sortedcontainers==2.4.0 soundfile==0.13.1 soxr==1.1.0 sqlitedict==2.1.0 sqlparse==0.5.5 sse-starlette==2.1.3 starlette==0.52.1 supervisor==4.3.0 SwissArmyTransformer==0.4.12 sympy==1.14.0 tabledata==1.3.5 tabulate==0.10.0 tcolorpy==0.1.7 tenacity==9.0.0 tensorboard==2.20.0 tensorboard-data-server==0.7.2 tensorboardX==2.6.5 tensordict==0.10.0 termcolor==3.3.0 tfrecord==1.14.6 threadpoolctl==3.6.0 tibs==0.5.7 tiktoken==0.7.0 tilelang==0.1.8 timm==1.0.11 tokenizers==0.22.2 toml==0.10.2 tomli==2.4.0 tomlkit==0.14.0 torch==2.10.0+cu128 torch_c_dlpack_ext==0.1.5 torchaudio==2.10.0+cu128 torchdata==0.11.0 torchmetrics==1.0.3 torchrec==1.5.0+cu128 torchvision==0.25.0+cu128 tqdm==4.67.3 tqdm-multiprocess==0.0.11 transformer_engine==2.11.0 transformer_engine_cu12==2.11.0 transformer_engine_torch==2.11.0 transformers==5.3.0 transformers-stream-generator==0.0.5 triton==3.6.0 trl==0.26.1 truststore @ file:///croot/truststore_1736550121485/work typepy==1.3.5 typer==0.23.1 typing-inspect==0.9.0 typing-inspection==0.4.2 typing_extensions==4.15.0 tyro==0.8.14 tzdata==2025.3 urllib3==2.7.0 uv==0.11.14 uvicorn==0.47.0 uvloop==0.21.0 # Editable Git install with no remote (verl==0.8.0.dev0) -e /opt/verl virtualenv==21.3.3 vllm==0.18.0 wandb==0.27.0 watchfiles==1.2.0 wcmatch==10.1 wcwidth==0.7.0 webdataset==1.0.2 websockets==16.0 Werkzeug==3.1.8 wheel==0.47.0 word2number==1.1 wrapt==1.17.3 xattr==1.3.0 xformers==0.0.35 xgrammar==0.1.33 xxhash==3.7.0 yapf==0.43.0 yarl==1.23.0 z3-solver==4.15.4.0 zipp==3.23.1 zstandard==0.25.0 ```

实验表现

测试样本
Ref 1 Ref 2
Image Image

Prompt:

为图1（人物图）模特双脚换上图2（鞋靴图）中的卡其色一字沙滩拖鞋，鞋款与脚部比例自然匹配，脚背与脚趾按凉拖开口露出，不穿袜子；替换后的双鞋须按图2呈现其鞋面的复古字母图案、红绿相间条纹和白
    色大底，并保留图2可见单侧的搭接结构与金色金属扣细节，未展示侧不强行臆造差异。图1中的腿部皮肤、黑色裤腿、手腕与黑色手表及灰色背景保持不变。
2K推理结果对比
  • 基于Flux2-klein-base-9B进行Lora微调，在step=100时，虽然迭代次数少，相比Base推理条件，图片中的噪音减少了；在step=8000时，右脚正常了。 能判断出Lora训练后在这个子任务上有改善。
  • 基于Qwen-image-2.1进行Lora微调，在step=100时，能看到模型推理效果变差，明显扭曲变形了。
基模名称 基模推理 Lora / Step=100 / Batch=8 Lora / Step=8000 / Batch=8
FLUX2-Klein-Base-9B Image Image Image
Qwen-Image-2.1 Image Image /

分析

  • 同样的数据，同样的方式进行训练，基于Flux2-klein-base-9B进行训练，在垂类业务上能看到明显改善；而基于Qwen-Image-2.1进行训练，模型推理效果快速下降，推测Qwen-Image-2.1是一个CFG蒸馏的模型，不适合进行常规方式微调。 这个Lora训练后模型推理效果快速下降问题，在蒸馏的Flux2-klein-9B模型上也遇到过。但是FLUX2官方文档进行了详细的说明，哪些是可以训练微调的，而Qwen-Image-2.1目前官方文档并没有明确提到这些。 不知道，我的猜测是否正确？
  • 基于Qwen-Image-2.1进行微调，其他人在其他垂类任务中也出现与我类似的问题,，微调后模型推理效果变差。 请问如何解决这些问题？ 目前看该代码仓库提供的基于Qwen-Image-2.1进行Lora微调是走不通的。

其他

  • FLUX2-Klein-Base-9B Lora训练脚步
点击展开/折叠 
accelerate launch examples/flux2/model_training/train.py \
  --dataset_base_path "" \
  --dataset_metadata_path shoes_train.json \
  --data_file_keys "image,edit_image" \
  --extra_inputs "edit_image" \
  --max_pixels 4194304 \
  --dataset_repeat 1 \
  --model_id_with_origin_paths "/weights/FLUX.2-klein-base-9B:text_encoder/*.safetensors,/weights/FLUX.2-klein-base-9B:transformer/*.safetensors,/weights/FLUX.2-klein-base-9B:vae/diffusion_pytorch_model.safetensors" \
  --tokenizer_path "/weights/FLUX.2-klein-base-9B:tokenizer/" \
  --learning_rate 1e-4 \
  --num_epochs 5 \
  --remove_prefix_in_ckpt "pipe.dit." \
  --output_path "./models/train/shoes/flux2" \
  --lora_base_model "dit" \
  --lora_target_modules "to_q,to_k,to_v,to_out.0,add_q_proj,add_k_proj,add_v_proj,to_add_out,linear_in,linear_out,to_qkv_mlp_proj,single_transformer_blocks.0.attn.to_out,single_transformer_blocks.1.attn.to_out,single_transformer_blocks.2.attn.to_out,single_transformer_blocks.3.attn.to_out,single_transformer_blocks.4.attn.to_out,single_transformer_blocks.5.attn.to_out,single_transformer_blocks.6.attn.to_out,single_transformer_blocks.7.attn.to_out,single_transformer_blocks.8.attn.to_out,single_transformer_blocks.9.attn.to_out,single_transformer_blocks.10.attn.to_out,single_transformer_blocks.11.attn.to_out,single_transformer_blocks.12.attn.to_out,single_transformer_blocks.13.attn.to_out,single_transformer_blocks.14.attn.to_out,single_transformer_blocks.15.attn.to_out,single_transformer_blocks.16.attn.to_out,single_transformer_blocks.17.attn.to_out,single_transformer_blocks.18.attn.to_out,single_transformer_blocks.19.attn.to_out,single_transformer_blocks.20.attn.to_out,single_transformer_blocks.21.attn.to_out,single_transformer_blocks.22.attn.to_out,single_transformer_blocks.23.attn.to_out" \
  --lora_rank 32 \
  --save_steps 1000 \
  --use_gradient_checkpointing
  • Qwen-Image-2.1 Lora训练脚步
点击展开/折叠 
export DIFFSYNTH_SKIP_DOWNLOAD=True
accelerate launch \
  examples/qwen_image_21/model_training/train.py \
  --dataset_base_path "" \
  --dataset_metadata_path shoes_train.json \
  --data_file_keys "image,edit_image" \
  --extra_inputs "edit_image" \
  --max_pixels 4194304 \
  --dataset_repeat 1 \
  --model_id_with_origin_paths "/weights/Qwen-Image-2.1:transformer/diffusion_pytorch_model*.safetensors,/weights/Qwen-Image-2.1:text_encoder/model*.safetensors,/weights/Qwen-Image-2.1:vae/diffusion_pytorch_model*.safetensors" \
  --processor_path "/weights/Qwen-Image-2.1/processor/" \
  --learning_rate 1e-4 \
  --num_epochs 20 \
  --remove_prefix_in_ckpt "pipe.dit." \
  --output_path "./models/train/Qwen-Image-2.1/" \
  --lora_base_model "dit" \
  --lora_target_modules "" \
  --lora_rank 32 \
  --use_gradient_checkpointing \
  --save_steps 100 \
  --find_unused_parameters
Linguagem predominante
Python
Estrelas
13.3k
Forks
1.3k
Merge médio
19h 4min
PRs com merge (30d)
40

Preparar o ambiente

Este projeto não oferece contêiner de desenvolvimento, Dockerfile nem guia de contribuição, então a configuração fica por sua conta: comece pelo README e veja nosso guia da primeira contribuição para os passos gerais.

Primeiros passos

  1. Leia a issue inteira e depois o guia de contribuição do projeto.
  2. Comente na issue dizendo que vai assumir — evita que duas pessoas façam o mesmo trabalho.
  3. Faça um fork do repositório e trabalhe em uma branch.
  4. Abra um pull request que referencie o número da issue.

Mais de modelscope/DiffSynth-Studio

Todas as issues de modelscope/DiffSynth-Studio

Issues semelhantes

Mais issues de Python

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.