huggingface/diffusers

Expanded init fields in StableDiffusionPipeline cause incompatibilities with many/most inherited pipelines

Open

#6,969 opened on Feb 13, 2024

View on GitHub
 (22 comments) (0 reactions) (1 assignee)Python (22,190 stars) (4,562 forks)batch import
bugcontributions-welcomegood first issuehelp wantedwip

Description

Describe the bug

class StableDiffusionPipeline has its init section as:

    def __init__(
        self,
        vae: AutoencoderKL,
        text_encoder: CLIPTextModel,
        tokenizer: CLIPTokenizer,
        unet: UNet2DConditionModel,
        scheduler: KarrasDiffusionSchedulers,
        safety_checker: StableDiffusionSafetyChecker,
        feature_extractor: CLIPImageProcessor,
        image_encoder: CLIPVisionModelWithProjection = None,
        requires_safety_checker: bool = True,
    ):

and here image_encoder was recently introduced thus changing the class signature
but most community pipelines do not include init field for recently included image_encoder and thus order or params is wrong

for example, examples/community/regional_prompting_stable_diffusion.py has this in its init:

        super().__init__(
            vae,
            text_encoder,
            tokenizer,
            unet,
            scheduler,
            safety_checker,
            feature_extractor,
            requires_safety_checker,
        )

which means bool value from requires_safety_checker is going to be passed as image_encoder and pipeline will fail during initialization like this:

> diffusers/pipelines/pipeline_utils.py:546 in _fetch_class_library_tuple
AttributeError: 'bool' object has no attribute '__module__'

this is a conceptual problem with changing master class signature while all inherited classes pass args list as simple list.
i don't see a simple solution as going back is bad and going forward requires updates to a lot of pipelines.

but at the very basic, at least add error handling to _fetch_class_library_tuple so invalid type does not cause entire solution to crash.

Reproduction

load nearly any community pipeline or any pipeline inherited from StableDiffusionPipeline, but not updated to use new signature in its super().__init__ call

Logs

No response

System Info

diffusers==0.26.3

Who can help?

@yiyixuxu @DN6 @sayakpaul @patrickvonplaten

Contributor guide