[training scripts] `--logit_mean` biases timesteps in the opposite direction to the SD3 paper (positive m oversamples low noise)
メンテナーはふだん 1 日以内に返信
@mlzerotoheroyoutube がすでに取り組んでいます。
2026年11月3日 から。
評価
- 難易度
- 4/5
- 見積もり時間
- 3〜5日
- 初心者へのやさしさ
- 35/100
- issue の種類
- バグ
- 明瞭さ
- 明確に書かれている
- 活発さ
- 活発
調査の方向性
Start with the sampling helper compute_density_for_timestep_sampling in src/diffusers/training_utils.py, then the indexing line in examples/dreambooth/train_dreambooth_lora_sd3.py that maps u onto noise_scheduler_copy.timesteps. Run the issue's reproduction script first to confirm the reversed E[sigma]. Done means the sign of logit_mean matches the SD3 paper across the 28 affected scripts, with the reproduction assertion passing. The maintainers still need to decide whether to flip the sampler or the indexing, since that changes behaviour for existing users.
索引モデルが issue の本文から書いたものです。
説明
Describe the bug
The SD3/FLUX-style training scripts sample timesteps with compute_density_for_timestep_sampling(weighting_scheme="logit_normal", logit_mean=m, logit_std=s), which returns u = sigmoid(N(m, s)). They then index the scheduler's timestep table with indices = (u * num_train_timesteps).long(). FlowMatchEulerDiscreteScheduler's table is ordered noise-first (timesteps[0] = 1000, sigmas[0] = 1), so the realised noise level is sigma ≈ shift(1 - u), i.e. sigma follows a (shifted) logit-normal(-m, s) distribution.
The SD3 paper (Esser et al. 2024, Sec. 3.1, Eq. 19), which the helper's docstring and README_sd3.md refer to, samples the noise level t of z_t = (1 - t) x_0 + t eps (the scripts' own comment: zt = (1 - texp) * x + texp * z1) from logit-normal(m, s), and says m biases training "towards either data p0 (negative m) or noise p1 (positive m)". In the scripts the sign of --logit_mean is therefore reversed: --logit_mean 1.0 --logit_std 0.6 (the paper's lognorm(1.00, 0.60)) concentrates training on low noise levels (FLUX: E[sigma] 0.284 instead of 0.717; P(sigma > 0.5) 0.048 instead of 0.952).
The defaults are not affected: logit_mean=0 and the mode / uniform schemes are symmetric. Only users who set --logit_mean get the opposite of what the paper describes.
Affected: the 28 scripts that use this indexing line (list below). Not affected:
train_dreambooth_lora_lumina2.py, which also reverses the interpolation ((1 - sigma) * noise + sigma * x0);train_dreambooth_lora_ideogram4.py, which uses the sampled value directly as sigma, i.e. the paper's direction.
Reproduction
This uses the real compute_density_for_timestep_sampling and FlowMatchEulerDiscreteScheduler with the FLUX.1 and SD3 scheduler configs, and the scripts' indexing:
# The SD3/FLUX training scripts sample timesteps like this (e.g. examples/dreambooth/train_dreambooth_sd3.py):
# u = compute_density_for_timestep_sampling(weighting_scheme="logit_normal", logit_mean=m, logit_std=s, ...)
# indices = (u * noise_scheduler_copy.config.num_train_timesteps).long()
# timesteps = noise_scheduler_copy.timesteps[indices] -> sigmas via get_sigmas(timesteps)
# noisy_model_input = (1.0 - sigmas) * model_input + sigmas * noise
# SD3 paper (Esser et al. 2024, Sec. 3.1, Eq. 19): t ~ logit-normal(m, s) is the noise level of
# z_t = (1 - t) x_0 + t eps, and m > 0 biases training towards noise. Here sigma plays the role of t.
import torch
import diffusers
from diffusers import FlowMatchEulerDiscreteScheduler
from diffusers.training_utils import compute_density_for_timestep_sampling
print(f"diffusers {diffusers.__version__}, torch {torch.__version__}")
m, s, n = 1.0, 0.6, 200_000
configs = { # scheduler configs of the FLUX.1 and SD3 checkpoints (scheduler/scheduler_config.json)
"FLUX.1": dict(shift=3.0, use_dynamic_shifting=True, base_shift=0.5, max_shift=1.15,
base_image_seq_len=256, max_image_seq_len=4096),
"SD3": dict(shift=3.0),
}
for name, cfg in configs.items():
sched = FlowMatchEulerDiscreteScheduler(num_train_timesteps=1000, **cfg)
torch.manual_seed(0)
u = compute_density_for_timestep_sampling(weighting_scheme="logit_normal", batch_size=n,
logit_mean=m, logit_std=s, mode_scale=1.29)
indices = (u * sched.config.num_train_timesteps).long() # as in the scripts
timesteps = sched.timesteps[indices]
# get_sigmas() looks up each timestep's position in the same table, i.e. `indices` (checked on a subset)
assert all((sched.timesteps == t).nonzero().item() == i for t, i in zip(timesteps[:500], indices[:500]))
sigma = sched.sigmas[indices]
# Paper direction: the noise level itself is logit-normal(m, s) (then the same table/shift)
sigma_paper = sched.sigmas[((1 - u) * 1000).long().clamp(max=999)]
print(f"{name:6s} logit_mean={m:+.1f}, logit_std={s}: E[sigma] scripts {sigma.mean():.3f}, paper {sigma_paper.mean():.3f}"
f" | P(sigma > 0.5) scripts {(sigma > 0.5).float().mean():.3f}, paper {(sigma_paper > 0.5).float().mean():.3f}")
assert sigma.mean() < 0.55 < sigma_paper.mean() # m > 0 should bias towards noise, the scripts bias towards data
print("BUG REPRODUCED: the sign of --logit_mean is reversed relative to the SD3 paper")
Logs
diffusers 0.42.1, torch 2.15.0+cpu
FLUX.1 logit_mean=+1.0, logit_std=0.6: E[sigma] scripts 0.284, paper 0.717 | P(sigma > 0.5) scripts 0.048, paper 0.952
SD3 logit_mean=+1.0, logit_std=0.6: E[sigma] scripts 0.524, paper 0.877 | P(sigma > 0.5) scripts 0.567, paper 1.000
BUG REPRODUCED: the sign of --logit_mean is reversed relative to the SD3 paper
Expected: with logit_mean=+1.0 the noise level should be biased towards noise, as in the paper (E[sigma] 0.717 for FLUX.1, 0.877 for SD3).
Where: training_utils.py#L366-L390 (v0.42.1, 2a14339) returns u, which correctly plays the role of the paper's t. The table is noise-first (scheduling_flow_match_euler_discrete.py#L127-L136). The scripts index it with u, e.g. train_dreambooth_lora_sd3.py#L1784-L1797:
u = compute_density_for_timestep_sampling(
weighting_scheme=args.weighting_scheme,
batch_size=bsz,
logit_mean=args.logit_mean,
logit_std=args.logit_std,
mode_scale=args.mode_scale,
)
indices = (u * noise_scheduler_copy.config.num_train_timesteps).long()
timesteps = noise_scheduler_copy.timesteps[indices].to(device=model_input.device)
# Add noise according to flow matching.
# zt = (1 - texp) * x + texp * z1
sigmas = get_sigmas(timesteps, n_dim=model_input.ndim, dtype=model_input.dtype)
noisy_model_input = (1.0 - sigmas) * model_input + sigmas * noise
The indexing line is identical, at the same line number, in all 28 scripts at v0.42.1 and on main (86ef420).
The 28 affected scripts (indexing line)
examples/advanced_diffusion_training/train_dreambooth_lora_flux_advanced.py#L2206examples/cogview4-control/train_control_cogview4.py#L1077examples/controlnet/train_controlnet_flux.py#L1288examples/controlnet/train_controlnet_sd3.py#L1320examples/dreambooth/train_dreambooth_flux.py#L1634examples/dreambooth/train_dreambooth_lora_bria_fibo2_img2img.py#L1731examples/dreambooth/train_dreambooth_lora_bria_fibo2.py#L1898examples/dreambooth/train_dreambooth_lora_flux2_img2img.py#L1779examples/dreambooth/train_dreambooth_lora_flux2_klein_img2img.py#L1725examples/dreambooth/train_dreambooth_lora_flux2_klein.py#L1845examples/dreambooth/train_dreambooth_lora_flux2.py#L1876examples/dreambooth/train_dreambooth_lora_flux3_klein_img2img.py#L1870examples/dreambooth/train_dreambooth_lora_flux3_klein.py#L1989examples/dreambooth/train_dreambooth_lora_flux_kontext.py#L2069examples/dreambooth/train_dreambooth_lora_flux.py#L1773examples/dreambooth/train_dreambooth_lora_hidream.py#L1597examples/dreambooth/train_dreambooth_lora_krea2.py#L1857examples/dreambooth/train_dreambooth_lora_qwenimage21_img2img.py#L1886examples/dreambooth/train_dreambooth_lora_qwenimage21.py#L1808examples/dreambooth/train_dreambooth_lora_qwen_image.py#L1753examples/dreambooth/train_dreambooth_lora_sana.py#L1410examples/dreambooth/train_dreambooth_lora_sd3.py#L1791examples/dreambooth/train_dreambooth_lora_z_image.py#L1741examples/dreambooth/train_dreambooth_sd3.py#L1587examples/flux-control/train_control_flux.py#L1067examples/flux-control/train_control_lora_flux.py#L1213examples/research_projects/flux_lora_quantization/train_dreambooth_lora_flux_miniature.py#L1063examples/research_projects/sd3_lora_colab/train_dreambooth_lora_sd3_miniature.py#L985
Proposed fix. The same one-line change in each of the 28 scripts. The shared helper cannot be flipped, because the ideogram4 script uses its output directly as sigma:
- indices = (u * noise_scheduler_copy.config.num_train_timesteps).long()
+ indices = ((1 - u) * noise_scheduler_copy.config.num_train_timesteps).long().clamp(
+ max=noise_scheduler_copy.config.num_train_timesteps - 1
+ )
The mode and uniform schemes, and logit_normal with logit_mean=0, are symmetric under u -> 1 - u, so their distributions do not change.
Verifying the fix. fix_check.py applies the fixed indexing with the same real helper and schedulers. It compares the realised sigmas with the paper's distribution sigma = shift(sigmoid(x)), x ~ N(m, s) (quantiles at 5/25/50/75/95%), and checks that the default settings keep the same distribution:
released = sched.sigmas[(u * N).long()]
fixed = sched.sigmas[((1 - u) * N).long().clamp(max=N - 1)]
t = torch.sigmoid(torch.randn(n) * s + m) # paper: the noise level t ~ logit-normal(m, s)
paper = shift * t / (1 + (shift - 1) * t) # shift = 1 for FLUX.1 (dynamic shifting), 3 for SD3
assert (fixed.quantile(qs) - paper.quantile(qs)).abs().max() < 5e-3
diffusers 0.42.1, torch 2.15.0+cpu
FLUX.1 logit_normal m=+1.0 s=0.6: E[sigma] released 0.284, fixed 0.717, paper 0.716 | max quantile gap fixed vs paper 0.0012
FLUX.1 logit_normal m=-0.5 s=1.0: E[sigma] released 0.603, fixed 0.398, paper 0.398 | max quantile gap fixed vs paper 0.0007
FLUX.1 logit_normal m=+0.0 s=1.0: E[sigma] released 0.501, fixed 0.500, paper 0.500 | max quantile gap fixed vs paper 0.0007 | max quantile gap fixed vs released 0.0020
FLUX.1 mode m=+0.0 s=1.0: E[sigma] released 0.501, fixed 0.500 | max quantile gap fixed vs released 0.0020
FLUX.1 none m=+0.0 s=1.0: E[sigma] released 0.500, fixed 0.501 | max quantile gap fixed vs released 0.0030
SD3 logit_normal m=+1.0 s=0.6: E[sigma] released 0.524, fixed 0.877, paper 0.877 | max quantile gap fixed vs paper 0.0009
SD3 logit_normal m=-0.5 s=1.0: E[sigma] released 0.794, fixed 0.621, paper 0.621 | max quantile gap fixed vs paper 0.0010
SD3 logit_normal m=+0.0 s=1.0: E[sigma] released 0.715, fixed 0.714, paper 0.714 | max quantile gap fixed vs paper 0.0011 | max quantile gap fixed vs released 0.0021
SD3 mode m=+0.0 s=1.0: E[sigma] released 0.710, fixed 0.710 | max quantile gap fixed vs released 0.0020
SD3 none m=+0.0 s=1.0: E[sigma] released 0.676, fixed 0.677 | max quantile gap fixed vs released 0.0027
FIX OK: the realised noise levels follow the SD3 paper's logit-normal(m, s); defaults unchanged
System Info
- 🤗 Diffusers version: 0.42.1
- Platform: Linux x86_64
- Python version: 3.13.13
- PyTorch version: 2.15.0+cpu (CPU only)
- Transformers version: 5.22.1
- Accelerate version: 1.15.0
Who can help?
No response
I would be happy to open a PR with this fix for the 28 scripts.
- 主要言語
- Python
- スター
- 34.9k
- フォーク
- 7.5k
- 平均マージ
- 3日 16時間
- マージ済み PR(30日)
- 65
環境構築
- Dockerfile・Docker Compose ファイルなし
- プルリクエストのテンプレートあり
- コントリビューションガイドを読む
はじめの一歩
- issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
- 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
- リポジトリをフォークし、ブランチを切って変更します。
- issue 番号を参照したプルリクエストを送ります。
huggingface/diffusers のほかの issue
-
難易度 2/5 1〜3時間 初心者へのやさしさ 84/100
huggingface/diffusers#15363 ·
メンテナーはふだん 1 日以内に返信
-
`ZImageImg2ImgPipeline` / `ZImageInpaintPipeline`: explicit height/width different from the input image raises RuntimeError対応中かも @chessing234 が 1 日前に担当しました。 オープンbug
難易度 2/5 1〜3時間 初心者へのやさしさ 70/100
huggingface/diffusers#15337 · コメント 2 件 ·
メンテナーはふだん 1 日以内に返信
-
Multi-vector textual inversion uses only the first vector when the token has uppercase letters (transformers v5)対応中かも @sunhouy が 2 日前に担当しました。 オープン
難易度 2/5 1〜3時間 初心者へのやさしさ 75/100
huggingface/diffusers#15313 ·
メンテナーはふだん 1 日以内に返信
-
TCDScheduler advances the generator on the final step of a strength-truncated custom schedule対応中かも @SIDDARTHAREDDY8 が 5 日前に担当しました。 オープンbug
難易度 2/5 1〜3時間 初心者へのやさしさ 74/100
huggingface/diffusers#15288 · コメント 2 件 ·
メンテナーはふだん 1 日以内に返信
-
unload_textual_inversion(token) leaves the extra tokens of a multi-vector embedding behind, so it can't be loaded again対応中かも @sohailshk が 11 日前に担当しました。 オープン
難易度 2/5 1〜3時間 初心者へのやさしさ 72/100
huggingface/diffusers#15211 · コメント 3 件 ·
メンテナーはふだん 1 日以内に返信
huggingface/diffusers の issue をすべて見る
似ている issue
-
難易度 2/5 1〜3時間 初心者へのやさしさ 73/100
メンテナーはふだん 1 日以内に返信
-
Unused loot bag group names left in dreamcraft en_US.lang対応中かも このイシューにリンクされたプルリクエストがオープン中、またはマージ済みです。 オープンStatus: Triage Suggestion
難易度 1/5 1時間未満 初心者へのやさしさ 90/100
GTNewHorizons/GT-New-Horizons-Modpack#27789 ·
メンテナーはふだん 1 日以内に返信
-
bug frontend sev-5
難易度 2/5 1〜3時間 初心者へのやさしさ 83/100
-
core good first issue
難易度 2/5 1〜3時間 初心者へのやさしさ 65/100
-
b4b code health next
難易度 2/5 1〜3時間 初心者へのやさしさ 68/100
メンテナーはふだん 1 日以内に返信