huggingface/diffusers

[Flux ControlNet] Add support for de-distilled models with CFG

Open

#9,635 opened on Oct 10, 2024

View on GitHub
 (11 comments) (0 reactions) (0 assignees)Python (4,562 forks)batch import
contributions-welcomegood first issue

Repository metrics

Stars
 (22,190 stars)
PR merge metrics
 (Avg merge 14d 10h) (101 merged PRs in 30d)

Description

Flux with Inpaiting and ControlNet currently yields bad result with the base model.

To echo this comment, using de-distilled models could potentially help getting better outputs.

Currently the Flux ControlNet pipelines do not support models like https://huggingface.co/nyanko7/flux-dev-de-distill

Contributor guide

Research direction
Examine the Flux ControlNet pipeline code to understand how models are loaded and CFG is applied. Investigate the de distilled model architecture differences and how to adapt the pipeline to handle them.
Tech stack
pythonpytorch
Domain
machine learningai
Issue type
Feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-2 days
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Mostly clear
Prerequisites
PythonPyTorchDiffusers
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
40