`StableDiffusionPipeline` produces `nans` for SD2.1 (Mac, cpu device)

Describe the bug

StableDiffusionPipeline produces nans for SD2.1.

The issue seems to be in the pipe.unet.conv_in. Printing the mean / std for sample before and after conv_in gives:

m,s = tensor(0.005) tensor(0.996)
m,s = tensor(15962061100191580160., grad_fn=<MeanBackward0>) tensor(16510648845702789070848., grad_fn=<StdBackward0>)

Reproduction

Make sure I'm using the latest official diffusers version

# uninstall local, editable version
!pip uninstall diffusers -y -qq
# install official version
!pip install diffusers -qq
# restart notebook now

Code to reproduce

from diffusers import StableDiffusionPipeline
pipe = StableDiffusionPipeline.from_pretrained('stabilityai/stable-diffusion-2-1')
result = pipe('the sun', output_type='pt')
result.images[0]

Output:

tensor([[[nan, nan, nan,  ..., nan, nan, nan],
         [nan, nan, nan,  ..., nan, nan, nan],
         [nan, nan, nan,  ..., nan, nan, nan],
...

Logs

No response

System Info

• diffusers version: 0.24.0
• Platform: macOS-14.1.2-arm64-arm-64bit
• Python version: 3.11.6
• PyTorch version (GPU?): 2.1.1 (False)
• Huggingface_hub version: 0.19.4
• Transformers version: 4.35.2
• Accelerate version: 0.25.0
• xFormers version: not installed
• Using GPU in script?: no
• Using distributed or parallel set-up in script?: no

Who can help?

Questions on pipelines > Stable Diffusion: @yiyixuxu @DN6 @sayakpaul @patrickvonplaten