Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

基于Qwen-Image-2.1进行Lora微调，效果很快变差，而同样的任务，基于FLUX2-Klein-base-9B微调效果在变好而没有变差，是什么问题？该如何解决？

Aperta
  • Qualcosa non va?
#1,756 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
15/100
Tipo di issue
Bug
Chiarezza
Da chiarire
Stato di attività
Attiva
Stack tecnologico
python
Ambito
machine-learning

Direzione di ricerca

The FLUX2 LoRA run is launched from examples/flux2/model_training/train.py, and the reporter's Qwen-Image-2.1 LoRA run is the one that degrades. Start by comparing the Qwen training entry point and its LoRA target modules against the FLUX2 script to find where the two diverge. Done would mean an explanation of the quality drop and a reproducible fix, which likely needs maintainer input on whether Qwen-Image-2.1 supports LoRA fine-tuning.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

设置

  • 代码 commit=39da7b9dbebd06af848b324a5b3a976c6d8e7839
  • 训练数据：3万条已经验证过的鞋靴穿搭多参考图数据
  • 测试数据： 1条样本
  • env 环境
点击展开/折叠查看依赖库及版本 ``` absl-py==2.4.0 accelerate==1.10.1 addict==2.4.0 aether==0.1.3.post1 aiobotocore==2.17.0 aiohappyeyeballs==2.6.1 aiohttp==3.13.5 aiohttp-cors==0.8.1 aioitertools==0.13.0 aiosignal==1.4.0 annotated-doc==0.0.4 annotated-types==0.7.0 anthropic==0.102.0 antlr4-python3-runtime==4.9.3 anyio==4.13.0 apache-tvm-ffi==0.1.8.post2 apex @ file:///opt/apex archspec @ file:///croot/archspec_1709217642129/work astor==0.8.1 astunparse==1.6.3 attrdict==2.0.1 attrs==26.1.0 audioread==3.1.0 autoawq==0.2.7.post3 autocommand==2.2.2 av==13.1.0 backports.tarfile==1.2.0 badam==1.2.2 bitarray==3.8.1 bitsandbytes==0.49.2 bitstring==4.4.0 blake3==1.0.8 blessed==1.39.0 blinker==1.9.0 boltons @ file:///croot/boltons_1737061692168/work boto3==1.35.54 botocore==1.35.93 braceexpand==0.1.7 bracex==2.6 brotli==1.2.0 cachetools==7.1.2 cbor2==6.1.1 certifi==2026.4.22 cffi==2.0.0 chardet==6.0.0.post1 charset-normalizer==3.4.7 click==8.4.0 cloudpickle==3.1.2 cmake==4.3.2 codetiming==1.4.0 colorama==0.4.6 colorful==0.5.8 compressed-tensors==0.13.0 conda @ file:///croot/conda_1738168370816/work conda-anaconda-telemetry @ file:///croot/conda-anaconda-telemetry_1736524596117/work conda-anaconda-tos @ file:///croot/conda-anaconda-tos_1739298990519/work conda-content-trust @ file:///croot/conda-content-trust_1714483159009/work conda-libmamba-solver @ file:///croot/conda-libmamba-solver_1737733694612/work/src conda-package-handling @ file:///croot/conda-package-handling_1731369017509/work conda_package_streaming @ file:///croot/conda-package-streaming_1731366181659/work confluent-kafka==2.13.2 contourpy==1.3.3 coverage==7.14.0 cpm-kernels==1.0.11 crc32c==2.8 cryptography==48.0.0 cuda-bindings==12.9.4 cuda-pathfinder==1.5.4 cuda-python==12.9.4 cupy-cuda12x==14.0.1 cycler==0.12.1 dacite==1.8.1 daft==0.7.13 DataProperty==1.1.1 datasets==4.8.4 decorator==5.3.1 decord==0.6.0 deep_ep==1.2.1 deepspeed==0.16.9 defusedxml==0.7.1 Deprecated==1.3.1 depyf==0.20.0 # Editable Git install with no remote (diffsynth==2.1.9) -e /home/ImageGen/DiffSynth-Studio-1013 diffusers==0.35.1 dill==0.4.1 diskcache==5.6.3 distlib==0.4.0 distro @ file:///croot/distro_1714488253808/work dnspython==2.8.0 docstring_parser==0.16 eametric==1.0.dev0 easydict==1.13 ebmlite==3.4.1 einops==0.8.0 elastic-transport==8.17.1 elasticsearch==8.15.1 email-validator==2.3.0 et_xmlfile==2.0.0 evaluate==0.4.6 fairscale==0.4.13 faiss-cpu==1.13.2 fastapi==0.133.1 fastapi-cli==0.0.24 fastapi-cloud-cli==0.17.1 fastar==0.11.0 fbgemm_gpu==1.5.0+cu128 filelock==3.29.0 filetype==1.2.0 fire==0.7.1 fla-core==0.5.0 FlagEmbedding==1.4.0 flash-linear-attention==0.5.0 flash_attn==2.7.4.post1 flashinfer-python==0.6.6 Flask==3.1.0 flask-cors==5.0.1 fonttools==4.63.0 frozendict @ file:///croot/frozendict_1713194832637/work frozenlist==1.8.0 fsspec==2026.1.0 ftfy==6.3.1 gguf==0.19.0 gitdb==4.0.12 GitPython==3.1.43 google-api-core==2.29.0 google-auth==2.53.0 googleapis-common-protos==1.73.0 gpustat==1.1.1 GPUtil==1.4.0 gradio==6.14.0 gradio_client==2.5.0 groovy==0.1.2 grpcio==1.80.0 grpcio-tools==1.80.0 h11==0.16.0 hf-gradio==0.4.1 hf-xet==1.5.0 hjson==3.1.0 httpcore==1.0.9 httptools==0.7.1 httpx==0.28.1 httpx-sse==0.4.3 huggingface_hub==1.15.0 hydra-core==1.3.2 idna==3.15 ijson==3.5.0 ImageIO==2.38.1 imageio-ffmpeg==0.6.0 importlib_metadata==8.7.1 iniconfig==2.3.0 interegular==0.3.3 iopath==0.1.10 itsdangerous==2.2.0 jaraco.context==6.1.0 jaraco.functools==4.4.0 jaraco.text==4.0.0 jieba==0.42.1 Jinja2==3.1.6 jiter==0.14.0 jmespath==1.1.0 joblib==1.5.3 jsonlines==4.0.0 jsonpatch @ file:///croot/jsonpatch_1714483231291/work jsonpointer==2.1 jsonschema==4.26.0 jsonschema-specifications==2025.9.1 kafka-python==2.0.2 kiwisolver==1.5.0 lance-namespace==0.7.6 lance-namespace-urllib3-client==0.7.6 lark==1.2.2 latex2mathml==3.81.0 lazy-loader==0.5 libmambapy @ file:///croot/mamba-split_1734469461757/work/libmambapy librosa==0.11.0 lightning-utilities==0.15.3 llguidance==1.3.0 llvmlite==0.44.0 lm-format-enforcer==0.11.3 lm_eval @ file:///opt/lm-evaluation-harness loguru==0.7.3 lxml==6.1.0 Markdown==3.10.2 markdown-it-py==4.2.0 MarkupSafe==3.0.3 mathruler==0.1.0 matplotlib==3.10.9 -e git+https://github.com/ISEEKYAN/mbridge.git@641a5a01de71080b2200d10e369090e40c9a351c#egg=mbridge mbstrdecoder==1.1.5 mcp==1.27.1 mdtex2html==1.3.0 mdurl==0.1.2 -e git+https://github.com/NVIDIA-NeMo/Megatron-Bridge.git@7bea48e09631a3fd97b7e08b91899aec949a4be5#egg=megatron_bridge # Editable Git install with no remote (megatron-core==0.16.0) -e /opt/Megatron-LM megatron-energon==6.0.1 menuinst @ file:///croot/menuinst_1738943416351/work mistral_common==1.11.2 ml_dtypes==0.5.4 mmengine==0.10.7 mmengine-lite==0.10.7 mmh3==5.0.1 model-hosting-container-standards==0.1.15 modelscope==1.40.2 modelscope-hub==0.4.6 more-itertools==11.0.2 mpi4py==4.1.2 mpmath==1.3.0 msgpack==1.1.2 msgspec==0.21.1 multi-storage-client==0.48.0 multidict==6.7.1 multiprocess==0.70.19 mypy_extensions==1.1.0 natsort==8.4.0 networkx==3.6.1 ninja==1.13.0 nltk==3.9.1 numba==0.61.2 numexpr==2.14.1 numpy==2.2.6 nvidia-cublas-cu12==12.8.4.1 nvidia-cuda-cupti-cu12==12.8.90 nvidia-cuda-nvrtc-cu12==12.8.93 nvidia-cuda-runtime-cu12==12.8.90 nvidia-cudnn-cu12==9.10.2.21 nvidia-cudnn-frontend==1.18.0 nvidia-cufft-cu12==11.3.3.83 nvidia-cufile-cu12==1.13.1.3 nvidia-curand-cu12==10.3.9.90 nvidia-cusolver-cu12==11.7.3.90 nvidia-cusparse-cu12==12.5.8.93 nvidia-cusparselt-cu12==0.7.1 nvidia-cutlass-dsl==4.4.2 nvidia-cutlass-dsl-libs-base==4.4.2 nvidia-ml-py==12.575.51 nvidia-modelopt==0.44.0 nvidia-nccl-cu12==2.27.5 nvidia-nvjitlink-cu12==12.8.93 nvidia-nvshmem-cu12==3.4.5 nvidia-nvtx-cu12==12.8.90 nvidia-resiliency-ext==0.6.0 nvtx==0.2.15 omegaconf==2.3.0 onnx==1.19.0 onnx-ir==0.2.1 onnxscript==0.7.0 openai==2.24.0 openai-harmony==0.0.4 opencensus==0.11.4 opencensus-context==0.1.3 opencv-python==4.10.0.84 opencv-python-headless==4.13.0.92 openpyxl==3.1.5 opentelemetry-api==1.41.1 opentelemetry-exporter-otlp==1.41.1 opentelemetry-exporter-otlp-proto-common==1.41.1 opentelemetry-exporter-otlp-proto-grpc==1.41.1 opentelemetry-exporter-otlp-proto-http==1.41.1 opentelemetry-exporter-prometheus==0.62b1 opentelemetry-proto==1.41.1 opentelemetry-sdk==1.41.1 opentelemetry-semantic-conventions==0.62b1 opentelemetry-semantic-conventions-ai==0.5.1 optimum==2.1.0 orjson==3.11.9 outlines_core==0.2.11 packaging==25.0 pandas==2.3.3 parameterized==0.9.0 partial-json-parser==0.2.1.1.post7 pathvalidate==3.3.1 peft==0.18.1 pillow==12.2.0 platformdirs==4.9.6 pluggy==1.6.0 pooch==1.9.0 portalocker==3.2.0 prettytable==3.17.0 prometheus-fastapi-instrumentator==7.1.0 prometheus_client==0.22.1 propcache==0.5.2 proto-plus==1.27.1 protobuf==5.29.6 psutil==7.2.2 PuLP==2.9.0 py-cpuinfo==9.0.0 py-spy==0.4.2 pyarrow==23.0.1 pyasn1==0.6.3 pyasn1_modules==0.4.2 pybase64==1.4.2 pybind11==3.0.4 pycosat @ file:///croot/pycosat_1736868416091/work pycountry==26.2.16 pycparser==3.0 pydantic==2.13.4 pydantic-extra-types==2.11.1 pydantic-settings==2.14.1 pydantic_core==2.46.4 pydub==0.25.1 Pygments==2.20.0 PyJWT==2.12.1 pylance==4.0.0 pylatexenc==2.10 pynvml==12.0.0 pyparsing==3.3.2 pyre-extensions==0.0.32 pytablewriter==1.2.1 pytest==8.4.1 pytest-asyncio==1.3.0 pytest-cov==6.2.1 pytest-html==4.1.1 pytest-metadata==3.1.1 python-dateutil==2.9.0.post0 python-discovery==1.3.1 python-dotenv==1.2.2 python-json-logger==4.1.0 python-multipart==0.0.29 pytz==2026.2 pyvers==0.1.0 PyYAML==6.0.3 pyzmq==27.1.0 quack-kernels==0.3.7 qwen-vl-utils==0.0.14 ray==2.51.1 referencing==0.37.0 regex==2026.5.9 requests==2.34.2 rich==15.0.0 rich-toolkit==0.19.9 rignore==0.7.6 rouge-chinese==1.0.3 rouge_score==0.1.2 rpds-py==0.30.0 ruff==0.14.6 s3fs==2026.1.0 s3transfer==0.10.4 sacrebleu==2.6.0 safehttpx==0.1.7 safetensors==0.7.0 scikit-build==0.19.0 scikit-learn==1.8.0 scipy==1.17.1 seaborn==0.13.2 semantic-version==2.10.0 sentence-transformers==5.2.3 sentencepiece==0.2.1 sentry-sdk==2.60.0 setproctitle==1.3.7 setuptools==80.10.2 shellingham==1.5.4 shortuuid==1.0.13 shtab==1.7.1 six==1.17.0 smart_open==7.6.1 smmap==5.0.3 sniffio==1.3.1 sortedcontainers==2.4.0 soundfile==0.13.1 soxr==1.1.0 sqlitedict==2.1.0 sqlparse==0.5.5 sse-starlette==2.1.3 starlette==0.52.1 supervisor==4.3.0 SwissArmyTransformer==0.4.12 sympy==1.14.0 tabledata==1.3.5 tabulate==0.10.0 tcolorpy==0.1.7 tenacity==9.0.0 tensorboard==2.20.0 tensorboard-data-server==0.7.2 tensorboardX==2.6.5 tensordict==0.10.0 termcolor==3.3.0 tfrecord==1.14.6 threadpoolctl==3.6.0 tibs==0.5.7 tiktoken==0.7.0 tilelang==0.1.8 timm==1.0.11 tokenizers==0.22.2 toml==0.10.2 tomli==2.4.0 tomlkit==0.14.0 torch==2.10.0+cu128 torch_c_dlpack_ext==0.1.5 torchaudio==2.10.0+cu128 torchdata==0.11.0 torchmetrics==1.0.3 torchrec==1.5.0+cu128 torchvision==0.25.0+cu128 tqdm==4.67.3 tqdm-multiprocess==0.0.11 transformer_engine==2.11.0 transformer_engine_cu12==2.11.0 transformer_engine_torch==2.11.0 transformers==5.3.0 transformers-stream-generator==0.0.5 triton==3.6.0 trl==0.26.1 truststore @ file:///croot/truststore_1736550121485/work typepy==1.3.5 typer==0.23.1 typing-inspect==0.9.0 typing-inspection==0.4.2 typing_extensions==4.15.0 tyro==0.8.14 tzdata==2025.3 urllib3==2.7.0 uv==0.11.14 uvicorn==0.47.0 uvloop==0.21.0 # Editable Git install with no remote (verl==0.8.0.dev0) -e /opt/verl virtualenv==21.3.3 vllm==0.18.0 wandb==0.27.0 watchfiles==1.2.0 wcmatch==10.1 wcwidth==0.7.0 webdataset==1.0.2 websockets==16.0 Werkzeug==3.1.8 wheel==0.47.0 word2number==1.1 wrapt==1.17.3 xattr==1.3.0 xformers==0.0.35 xgrammar==0.1.33 xxhash==3.7.0 yapf==0.43.0 yarl==1.23.0 z3-solver==4.15.4.0 zipp==3.23.1 zstandard==0.25.0 ```

实验表现

测试样本
Ref 1 Ref 2
Image Image

Prompt:

为图1（人物图）模特双脚换上图2（鞋靴图）中的卡其色一字沙滩拖鞋，鞋款与脚部比例自然匹配，脚背与脚趾按凉拖开口露出，不穿袜子；替换后的双鞋须按图2呈现其鞋面的复古字母图案、红绿相间条纹和白
    色大底，并保留图2可见单侧的搭接结构与金色金属扣细节，未展示侧不强行臆造差异。图1中的腿部皮肤、黑色裤腿、手腕与黑色手表及灰色背景保持不变。
2K推理结果对比
  • 基于Flux2-klein-base-9B进行Lora微调，在step=100时，虽然迭代次数少，相比Base推理条件，图片中的噪音减少了；在step=8000时，右脚正常了。 能判断出Lora训练后在这个子任务上有改善。
  • 基于Qwen-image-2.1进行Lora微调，在step=100时，能看到模型推理效果变差，明显扭曲变形了。
基模名称 基模推理 Lora / Step=100 / Batch=8 Lora / Step=8000 / Batch=8
FLUX2-Klein-Base-9B Image Image Image
Qwen-Image-2.1 Image Image /

分析

  • 同样的数据，同样的方式进行训练，基于Flux2-klein-base-9B进行训练，在垂类业务上能看到明显改善；而基于Qwen-Image-2.1进行训练，模型推理效果快速下降，推测Qwen-Image-2.1是一个CFG蒸馏的模型，不适合进行常规方式微调。 这个Lora训练后模型推理效果快速下降问题，在蒸馏的Flux2-klein-9B模型上也遇到过。但是FLUX2官方文档进行了详细的说明，哪些是可以训练微调的，而Qwen-Image-2.1目前官方文档并没有明确提到这些。 不知道，我的猜测是否正确？
  • 基于Qwen-Image-2.1进行微调，其他人在其他垂类任务中也出现与我类似的问题,，微调后模型推理效果变差。 请问如何解决这些问题？ 目前看该代码仓库提供的基于Qwen-Image-2.1进行Lora微调是走不通的。

其他

  • FLUX2-Klein-Base-9B Lora训练脚步
点击展开/折叠 
accelerate launch examples/flux2/model_training/train.py \
  --dataset_base_path "" \
  --dataset_metadata_path shoes_train.json \
  --data_file_keys "image,edit_image" \
  --extra_inputs "edit_image" \
  --max_pixels 4194304 \
  --dataset_repeat 1 \
  --model_id_with_origin_paths "/weights/FLUX.2-klein-base-9B:text_encoder/*.safetensors,/weights/FLUX.2-klein-base-9B:transformer/*.safetensors,/weights/FLUX.2-klein-base-9B:vae/diffusion_pytorch_model.safetensors" \
  --tokenizer_path "/weights/FLUX.2-klein-base-9B:tokenizer/" \
  --learning_rate 1e-4 \
  --num_epochs 5 \
  --remove_prefix_in_ckpt "pipe.dit." \
  --output_path "./models/train/shoes/flux2" \
  --lora_base_model "dit" \
  --lora_target_modules "to_q,to_k,to_v,to_out.0,add_q_proj,add_k_proj,add_v_proj,to_add_out,linear_in,linear_out,to_qkv_mlp_proj,single_transformer_blocks.0.attn.to_out,single_transformer_blocks.1.attn.to_out,single_transformer_blocks.2.attn.to_out,single_transformer_blocks.3.attn.to_out,single_transformer_blocks.4.attn.to_out,single_transformer_blocks.5.attn.to_out,single_transformer_blocks.6.attn.to_out,single_transformer_blocks.7.attn.to_out,single_transformer_blocks.8.attn.to_out,single_transformer_blocks.9.attn.to_out,single_transformer_blocks.10.attn.to_out,single_transformer_blocks.11.attn.to_out,single_transformer_blocks.12.attn.to_out,single_transformer_blocks.13.attn.to_out,single_transformer_blocks.14.attn.to_out,single_transformer_blocks.15.attn.to_out,single_transformer_blocks.16.attn.to_out,single_transformer_blocks.17.attn.to_out,single_transformer_blocks.18.attn.to_out,single_transformer_blocks.19.attn.to_out,single_transformer_blocks.20.attn.to_out,single_transformer_blocks.21.attn.to_out,single_transformer_blocks.22.attn.to_out,single_transformer_blocks.23.attn.to_out" \
  --lora_rank 32 \
  --save_steps 1000 \
  --use_gradient_checkpointing
  • Qwen-Image-2.1 Lora训练脚步
点击展开/折叠 
export DIFFSYNTH_SKIP_DOWNLOAD=True
accelerate launch \
  examples/qwen_image_21/model_training/train.py \
  --dataset_base_path "" \
  --dataset_metadata_path shoes_train.json \
  --data_file_keys "image,edit_image" \
  --extra_inputs "edit_image" \
  --max_pixels 4194304 \
  --dataset_repeat 1 \
  --model_id_with_origin_paths "/weights/Qwen-Image-2.1:transformer/diffusion_pytorch_model*.safetensors,/weights/Qwen-Image-2.1:text_encoder/model*.safetensors,/weights/Qwen-Image-2.1:vae/diffusion_pytorch_model*.safetensors" \
  --processor_path "/weights/Qwen-Image-2.1/processor/" \
  --learning_rate 1e-4 \
  --num_epochs 20 \
  --remove_prefix_in_ckpt "pipe.dit." \
  --output_path "./models/train/Qwen-Image-2.1/" \
  --lora_base_model "dit" \
  --lora_target_modules "" \
  --lora_rank 32 \
  --use_gradient_checkpointing \
  --save_steps 100 \
  --find_unused_parameters
Lingua principale
Python
Stelle
13.2k
Fork
1.3k
Merge medio
3g 17h
PR unite (30g)
32

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di modelscope/DiffSynth-Studio

Tutte le issue di modelscope/DiffSynth-Studio

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.