Hacktoberfest 2026: die Issues, die Maintainer für den Oktober markiert haben – offen und einsteigerfreundlich. Hacktoberfest-Issues durchsuchen

基于Qwen-Image-2.1进行Lora微调，效果很快变差，而同样的任务，基于FLUX2-Klein-base-9B微调效果在变好而没有变差，是什么问题？该如何解决？

Offen
  • Stimmt etwas nicht?
#1,756 1 Kommentar 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen

Maintainer antworten meist innerhalb von 1 Tag

Dieses Issue hat noch niemand übernommen.

Bewertung

Schwierigkeit
5/5
Geschätzter Aufwand
Über eine Woche
Anfängerfreundlichkeit
15/100
Issue-Typ
Bug
Klarheit
Muss geklärt werden
Aktivitätsstatus
Aktiv
Tech-Stack
python
Bereich
machine-learning

Rechercherichtung

The FLUX2 LoRA run is launched from examples/flux2/model_training/train.py, and the reporter's Qwen-Image-2.1 LoRA run is the one that degrades. Start by comparing the Qwen training entry point and its LoRA target modules against the FLUX2 script to find where the two diverge. Done would mean an explanation of the quality drop and a reproducible fix, which likely needs maintainer input on whether Qwen-Image-2.1 supports LoRA fine-tuning.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Beschreibung

设置

  • 代码 commit=39da7b9dbebd06af848b324a5b3a976c6d8e7839
  • 训练数据：3万条已经验证过的鞋靴穿搭多参考图数据
  • 测试数据： 1条样本
  • env 环境
点击展开/折叠查看依赖库及版本 ``` absl-py==2.4.0 accelerate==1.10.1 addict==2.4.0 aether==0.1.3.post1 aiobotocore==2.17.0 aiohappyeyeballs==2.6.1 aiohttp==3.13.5 aiohttp-cors==0.8.1 aioitertools==0.13.0 aiosignal==1.4.0 annotated-doc==0.0.4 annotated-types==0.7.0 anthropic==0.102.0 antlr4-python3-runtime==4.9.3 anyio==4.13.0 apache-tvm-ffi==0.1.8.post2 apex @ file:///opt/apex archspec @ file:///croot/archspec_1709217642129/work astor==0.8.1 astunparse==1.6.3 attrdict==2.0.1 attrs==26.1.0 audioread==3.1.0 autoawq==0.2.7.post3 autocommand==2.2.2 av==13.1.0 backports.tarfile==1.2.0 badam==1.2.2 bitarray==3.8.1 bitsandbytes==0.49.2 bitstring==4.4.0 blake3==1.0.8 blessed==1.39.0 blinker==1.9.0 boltons @ file:///croot/boltons_1737061692168/work boto3==1.35.54 botocore==1.35.93 braceexpand==0.1.7 bracex==2.6 brotli==1.2.0 cachetools==7.1.2 cbor2==6.1.1 certifi==2026.4.22 cffi==2.0.0 chardet==6.0.0.post1 charset-normalizer==3.4.7 click==8.4.0 cloudpickle==3.1.2 cmake==4.3.2 codetiming==1.4.0 colorama==0.4.6 colorful==0.5.8 compressed-tensors==0.13.0 conda @ file:///croot/conda_1738168370816/work conda-anaconda-telemetry @ file:///croot/conda-anaconda-telemetry_1736524596117/work conda-anaconda-tos @ file:///croot/conda-anaconda-tos_1739298990519/work conda-content-trust @ file:///croot/conda-content-trust_1714483159009/work conda-libmamba-solver @ file:///croot/conda-libmamba-solver_1737733694612/work/src conda-package-handling @ file:///croot/conda-package-handling_1731369017509/work conda_package_streaming @ file:///croot/conda-package-streaming_1731366181659/work confluent-kafka==2.13.2 contourpy==1.3.3 coverage==7.14.0 cpm-kernels==1.0.11 crc32c==2.8 cryptography==48.0.0 cuda-bindings==12.9.4 cuda-pathfinder==1.5.4 cuda-python==12.9.4 cupy-cuda12x==14.0.1 cycler==0.12.1 dacite==1.8.1 daft==0.7.13 DataProperty==1.1.1 datasets==4.8.4 decorator==5.3.1 decord==0.6.0 deep_ep==1.2.1 deepspeed==0.16.9 defusedxml==0.7.1 Deprecated==1.3.1 depyf==0.20.0 # Editable Git install with no remote (diffsynth==2.1.9) -e /home/ImageGen/DiffSynth-Studio-1013 diffusers==0.35.1 dill==0.4.1 diskcache==5.6.3 distlib==0.4.0 distro @ file:///croot/distro_1714488253808/work dnspython==2.8.0 docstring_parser==0.16 eametric==1.0.dev0 easydict==1.13 ebmlite==3.4.1 einops==0.8.0 elastic-transport==8.17.1 elasticsearch==8.15.1 email-validator==2.3.0 et_xmlfile==2.0.0 evaluate==0.4.6 fairscale==0.4.13 faiss-cpu==1.13.2 fastapi==0.133.1 fastapi-cli==0.0.24 fastapi-cloud-cli==0.17.1 fastar==0.11.0 fbgemm_gpu==1.5.0+cu128 filelock==3.29.0 filetype==1.2.0 fire==0.7.1 fla-core==0.5.0 FlagEmbedding==1.4.0 flash-linear-attention==0.5.0 flash_attn==2.7.4.post1 flashinfer-python==0.6.6 Flask==3.1.0 flask-cors==5.0.1 fonttools==4.63.0 frozendict @ file:///croot/frozendict_1713194832637/work frozenlist==1.8.0 fsspec==2026.1.0 ftfy==6.3.1 gguf==0.19.0 gitdb==4.0.12 GitPython==3.1.43 google-api-core==2.29.0 google-auth==2.53.0 googleapis-common-protos==1.73.0 gpustat==1.1.1 GPUtil==1.4.0 gradio==6.14.0 gradio_client==2.5.0 groovy==0.1.2 grpcio==1.80.0 grpcio-tools==1.80.0 h11==0.16.0 hf-gradio==0.4.1 hf-xet==1.5.0 hjson==3.1.0 httpcore==1.0.9 httptools==0.7.1 httpx==0.28.1 httpx-sse==0.4.3 huggingface_hub==1.15.0 hydra-core==1.3.2 idna==3.15 ijson==3.5.0 ImageIO==2.38.1 imageio-ffmpeg==0.6.0 importlib_metadata==8.7.1 iniconfig==2.3.0 interegular==0.3.3 iopath==0.1.10 itsdangerous==2.2.0 jaraco.context==6.1.0 jaraco.functools==4.4.0 jaraco.text==4.0.0 jieba==0.42.1 Jinja2==3.1.6 jiter==0.14.0 jmespath==1.1.0 joblib==1.5.3 jsonlines==4.0.0 jsonpatch @ file:///croot/jsonpatch_1714483231291/work jsonpointer==2.1 jsonschema==4.26.0 jsonschema-specifications==2025.9.1 kafka-python==2.0.2 kiwisolver==1.5.0 lance-namespace==0.7.6 lance-namespace-urllib3-client==0.7.6 lark==1.2.2 latex2mathml==3.81.0 lazy-loader==0.5 libmambapy @ file:///croot/mamba-split_1734469461757/work/libmambapy librosa==0.11.0 lightning-utilities==0.15.3 llguidance==1.3.0 llvmlite==0.44.0 lm-format-enforcer==0.11.3 lm_eval @ file:///opt/lm-evaluation-harness loguru==0.7.3 lxml==6.1.0 Markdown==3.10.2 markdown-it-py==4.2.0 MarkupSafe==3.0.3 mathruler==0.1.0 matplotlib==3.10.9 -e git+https://github.com/ISEEKYAN/mbridge.git@641a5a01de71080b2200d10e369090e40c9a351c#egg=mbridge mbstrdecoder==1.1.5 mcp==1.27.1 mdtex2html==1.3.0 mdurl==0.1.2 -e git+https://github.com/NVIDIA-NeMo/Megatron-Bridge.git@7bea48e09631a3fd97b7e08b91899aec949a4be5#egg=megatron_bridge # Editable Git install with no remote (megatron-core==0.16.0) -e /opt/Megatron-LM megatron-energon==6.0.1 menuinst @ file:///croot/menuinst_1738943416351/work mistral_common==1.11.2 ml_dtypes==0.5.4 mmengine==0.10.7 mmengine-lite==0.10.7 mmh3==5.0.1 model-hosting-container-standards==0.1.15 modelscope==1.40.2 modelscope-hub==0.4.6 more-itertools==11.0.2 mpi4py==4.1.2 mpmath==1.3.0 msgpack==1.1.2 msgspec==0.21.1 multi-storage-client==0.48.0 multidict==6.7.1 multiprocess==0.70.19 mypy_extensions==1.1.0 natsort==8.4.0 networkx==3.6.1 ninja==1.13.0 nltk==3.9.1 numba==0.61.2 numexpr==2.14.1 numpy==2.2.6 nvidia-cublas-cu12==12.8.4.1 nvidia-cuda-cupti-cu12==12.8.90 nvidia-cuda-nvrtc-cu12==12.8.93 nvidia-cuda-runtime-cu12==12.8.90 nvidia-cudnn-cu12==9.10.2.21 nvidia-cudnn-frontend==1.18.0 nvidia-cufft-cu12==11.3.3.83 nvidia-cufile-cu12==1.13.1.3 nvidia-curand-cu12==10.3.9.90 nvidia-cusolver-cu12==11.7.3.90 nvidia-cusparse-cu12==12.5.8.93 nvidia-cusparselt-cu12==0.7.1 nvidia-cutlass-dsl==4.4.2 nvidia-cutlass-dsl-libs-base==4.4.2 nvidia-ml-py==12.575.51 nvidia-modelopt==0.44.0 nvidia-nccl-cu12==2.27.5 nvidia-nvjitlink-cu12==12.8.93 nvidia-nvshmem-cu12==3.4.5 nvidia-nvtx-cu12==12.8.90 nvidia-resiliency-ext==0.6.0 nvtx==0.2.15 omegaconf==2.3.0 onnx==1.19.0 onnx-ir==0.2.1 onnxscript==0.7.0 openai==2.24.0 openai-harmony==0.0.4 opencensus==0.11.4 opencensus-context==0.1.3 opencv-python==4.10.0.84 opencv-python-headless==4.13.0.92 openpyxl==3.1.5 opentelemetry-api==1.41.1 opentelemetry-exporter-otlp==1.41.1 opentelemetry-exporter-otlp-proto-common==1.41.1 opentelemetry-exporter-otlp-proto-grpc==1.41.1 opentelemetry-exporter-otlp-proto-http==1.41.1 opentelemetry-exporter-prometheus==0.62b1 opentelemetry-proto==1.41.1 opentelemetry-sdk==1.41.1 opentelemetry-semantic-conventions==0.62b1 opentelemetry-semantic-conventions-ai==0.5.1 optimum==2.1.0 orjson==3.11.9 outlines_core==0.2.11 packaging==25.0 pandas==2.3.3 parameterized==0.9.0 partial-json-parser==0.2.1.1.post7 pathvalidate==3.3.1 peft==0.18.1 pillow==12.2.0 platformdirs==4.9.6 pluggy==1.6.0 pooch==1.9.0 portalocker==3.2.0 prettytable==3.17.0 prometheus-fastapi-instrumentator==7.1.0 prometheus_client==0.22.1 propcache==0.5.2 proto-plus==1.27.1 protobuf==5.29.6 psutil==7.2.2 PuLP==2.9.0 py-cpuinfo==9.0.0 py-spy==0.4.2 pyarrow==23.0.1 pyasn1==0.6.3 pyasn1_modules==0.4.2 pybase64==1.4.2 pybind11==3.0.4 pycosat @ file:///croot/pycosat_1736868416091/work pycountry==26.2.16 pycparser==3.0 pydantic==2.13.4 pydantic-extra-types==2.11.1 pydantic-settings==2.14.1 pydantic_core==2.46.4 pydub==0.25.1 Pygments==2.20.0 PyJWT==2.12.1 pylance==4.0.0 pylatexenc==2.10 pynvml==12.0.0 pyparsing==3.3.2 pyre-extensions==0.0.32 pytablewriter==1.2.1 pytest==8.4.1 pytest-asyncio==1.3.0 pytest-cov==6.2.1 pytest-html==4.1.1 pytest-metadata==3.1.1 python-dateutil==2.9.0.post0 python-discovery==1.3.1 python-dotenv==1.2.2 python-json-logger==4.1.0 python-multipart==0.0.29 pytz==2026.2 pyvers==0.1.0 PyYAML==6.0.3 pyzmq==27.1.0 quack-kernels==0.3.7 qwen-vl-utils==0.0.14 ray==2.51.1 referencing==0.37.0 regex==2026.5.9 requests==2.34.2 rich==15.0.0 rich-toolkit==0.19.9 rignore==0.7.6 rouge-chinese==1.0.3 rouge_score==0.1.2 rpds-py==0.30.0 ruff==0.14.6 s3fs==2026.1.0 s3transfer==0.10.4 sacrebleu==2.6.0 safehttpx==0.1.7 safetensors==0.7.0 scikit-build==0.19.0 scikit-learn==1.8.0 scipy==1.17.1 seaborn==0.13.2 semantic-version==2.10.0 sentence-transformers==5.2.3 sentencepiece==0.2.1 sentry-sdk==2.60.0 setproctitle==1.3.7 setuptools==80.10.2 shellingham==1.5.4 shortuuid==1.0.13 shtab==1.7.1 six==1.17.0 smart_open==7.6.1 smmap==5.0.3 sniffio==1.3.1 sortedcontainers==2.4.0 soundfile==0.13.1 soxr==1.1.0 sqlitedict==2.1.0 sqlparse==0.5.5 sse-starlette==2.1.3 starlette==0.52.1 supervisor==4.3.0 SwissArmyTransformer==0.4.12 sympy==1.14.0 tabledata==1.3.5 tabulate==0.10.0 tcolorpy==0.1.7 tenacity==9.0.0 tensorboard==2.20.0 tensorboard-data-server==0.7.2 tensorboardX==2.6.5 tensordict==0.10.0 termcolor==3.3.0 tfrecord==1.14.6 threadpoolctl==3.6.0 tibs==0.5.7 tiktoken==0.7.0 tilelang==0.1.8 timm==1.0.11 tokenizers==0.22.2 toml==0.10.2 tomli==2.4.0 tomlkit==0.14.0 torch==2.10.0+cu128 torch_c_dlpack_ext==0.1.5 torchaudio==2.10.0+cu128 torchdata==0.11.0 torchmetrics==1.0.3 torchrec==1.5.0+cu128 torchvision==0.25.0+cu128 tqdm==4.67.3 tqdm-multiprocess==0.0.11 transformer_engine==2.11.0 transformer_engine_cu12==2.11.0 transformer_engine_torch==2.11.0 transformers==5.3.0 transformers-stream-generator==0.0.5 triton==3.6.0 trl==0.26.1 truststore @ file:///croot/truststore_1736550121485/work typepy==1.3.5 typer==0.23.1 typing-inspect==0.9.0 typing-inspection==0.4.2 typing_extensions==4.15.0 tyro==0.8.14 tzdata==2025.3 urllib3==2.7.0 uv==0.11.14 uvicorn==0.47.0 uvloop==0.21.0 # Editable Git install with no remote (verl==0.8.0.dev0) -e /opt/verl virtualenv==21.3.3 vllm==0.18.0 wandb==0.27.0 watchfiles==1.2.0 wcmatch==10.1 wcwidth==0.7.0 webdataset==1.0.2 websockets==16.0 Werkzeug==3.1.8 wheel==0.47.0 word2number==1.1 wrapt==1.17.3 xattr==1.3.0 xformers==0.0.35 xgrammar==0.1.33 xxhash==3.7.0 yapf==0.43.0 yarl==1.23.0 z3-solver==4.15.4.0 zipp==3.23.1 zstandard==0.25.0 ```

实验表现

测试样本
Ref 1 Ref 2
Image Image

Prompt:

为图1（人物图）模特双脚换上图2（鞋靴图）中的卡其色一字沙滩拖鞋，鞋款与脚部比例自然匹配，脚背与脚趾按凉拖开口露出，不穿袜子；替换后的双鞋须按图2呈现其鞋面的复古字母图案、红绿相间条纹和白
    色大底，并保留图2可见单侧的搭接结构与金色金属扣细节，未展示侧不强行臆造差异。图1中的腿部皮肤、黑色裤腿、手腕与黑色手表及灰色背景保持不变。
2K推理结果对比
  • 基于Flux2-klein-base-9B进行Lora微调，在step=100时，虽然迭代次数少，相比Base推理条件，图片中的噪音减少了；在step=8000时，右脚正常了。 能判断出Lora训练后在这个子任务上有改善。
  • 基于Qwen-image-2.1进行Lora微调，在step=100时，能看到模型推理效果变差，明显扭曲变形了。
基模名称 基模推理 Lora / Step=100 / Batch=8 Lora / Step=8000 / Batch=8
FLUX2-Klein-Base-9B Image Image Image
Qwen-Image-2.1 Image Image /

分析

  • 同样的数据，同样的方式进行训练，基于Flux2-klein-base-9B进行训练，在垂类业务上能看到明显改善；而基于Qwen-Image-2.1进行训练，模型推理效果快速下降，推测Qwen-Image-2.1是一个CFG蒸馏的模型，不适合进行常规方式微调。 这个Lora训练后模型推理效果快速下降问题，在蒸馏的Flux2-klein-9B模型上也遇到过。但是FLUX2官方文档进行了详细的说明，哪些是可以训练微调的，而Qwen-Image-2.1目前官方文档并没有明确提到这些。 不知道，我的猜测是否正确？
  • 基于Qwen-Image-2.1进行微调，其他人在其他垂类任务中也出现与我类似的问题,，微调后模型推理效果变差。 请问如何解决这些问题？ 目前看该代码仓库提供的基于Qwen-Image-2.1进行Lora微调是走不通的。

其他

  • FLUX2-Klein-Base-9B Lora训练脚步
点击展开/折叠 
accelerate launch examples/flux2/model_training/train.py \
  --dataset_base_path "" \
  --dataset_metadata_path shoes_train.json \
  --data_file_keys "image,edit_image" \
  --extra_inputs "edit_image" \
  --max_pixels 4194304 \
  --dataset_repeat 1 \
  --model_id_with_origin_paths "/weights/FLUX.2-klein-base-9B:text_encoder/*.safetensors,/weights/FLUX.2-klein-base-9B:transformer/*.safetensors,/weights/FLUX.2-klein-base-9B:vae/diffusion_pytorch_model.safetensors" \
  --tokenizer_path "/weights/FLUX.2-klein-base-9B:tokenizer/" \
  --learning_rate 1e-4 \
  --num_epochs 5 \
  --remove_prefix_in_ckpt "pipe.dit." \
  --output_path "./models/train/shoes/flux2" \
  --lora_base_model "dit" \
  --lora_target_modules "to_q,to_k,to_v,to_out.0,add_q_proj,add_k_proj,add_v_proj,to_add_out,linear_in,linear_out,to_qkv_mlp_proj,single_transformer_blocks.0.attn.to_out,single_transformer_blocks.1.attn.to_out,single_transformer_blocks.2.attn.to_out,single_transformer_blocks.3.attn.to_out,single_transformer_blocks.4.attn.to_out,single_transformer_blocks.5.attn.to_out,single_transformer_blocks.6.attn.to_out,single_transformer_blocks.7.attn.to_out,single_transformer_blocks.8.attn.to_out,single_transformer_blocks.9.attn.to_out,single_transformer_blocks.10.attn.to_out,single_transformer_blocks.11.attn.to_out,single_transformer_blocks.12.attn.to_out,single_transformer_blocks.13.attn.to_out,single_transformer_blocks.14.attn.to_out,single_transformer_blocks.15.attn.to_out,single_transformer_blocks.16.attn.to_out,single_transformer_blocks.17.attn.to_out,single_transformer_blocks.18.attn.to_out,single_transformer_blocks.19.attn.to_out,single_transformer_blocks.20.attn.to_out,single_transformer_blocks.21.attn.to_out,single_transformer_blocks.22.attn.to_out,single_transformer_blocks.23.attn.to_out" \
  --lora_rank 32 \
  --save_steps 1000 \
  --use_gradient_checkpointing
  • Qwen-Image-2.1 Lora训练脚步
点击展开/折叠 
export DIFFSYNTH_SKIP_DOWNLOAD=True
accelerate launch \
  examples/qwen_image_21/model_training/train.py \
  --dataset_base_path "" \
  --dataset_metadata_path shoes_train.json \
  --data_file_keys "image,edit_image" \
  --extra_inputs "edit_image" \
  --max_pixels 4194304 \
  --dataset_repeat 1 \
  --model_id_with_origin_paths "/weights/Qwen-Image-2.1:transformer/diffusion_pytorch_model*.safetensors,/weights/Qwen-Image-2.1:text_encoder/model*.safetensors,/weights/Qwen-Image-2.1:vae/diffusion_pytorch_model*.safetensors" \
  --processor_path "/weights/Qwen-Image-2.1/processor/" \
  --learning_rate 1e-4 \
  --num_epochs 20 \
  --remove_prefix_in_ckpt "pipe.dit." \
  --output_path "./models/train/Qwen-Image-2.1/" \
  --lora_base_model "dit" \
  --lora_target_modules "" \
  --lora_rank 32 \
  --use_gradient_checkpointing \
  --save_steps 100 \
  --find_unused_parameters
Vorherrschende Sprache
Python
Sterne
13.2k
Forks
1.3k
Ø Merge
3 T. 17 Std.
Gemergte PRs (30 T.)
32

Entwicklungsumgebung

Dieses Projekt bietet weder Dev-Container noch Dockerfile noch Beitragsleitfaden – die Einrichtung liegt bei Ihnen. Beginnen Sie mit der README; die allgemeinen Schritte stehen in unserem Leitfaden für den ersten Beitrag.

Erste Schritte

  1. Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
  2. Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
  3. Forken Sie das Repository und arbeiten Sie in einem Branch.
  4. Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.

Mehr aus modelscope/DiffSynth-Studio

Alle Issues in modelscope/DiffSynth-Studio

Ähnliche Issues

Weitere Issues zu Python

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.