YuE2 VAE: a grown decoder (YuE2-VAE-RS x3) loses 210 tensors silently under strict=False; a 35-line patch is ready

Aperta
  • Qualcosa non va?
#17,697 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
43/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
python, pytorch
Ambito
backend, machine-learning

Direzione di ricerca

Read comfy/ldm/audio/autoencoder.py and the YuE2 branch in comfy/sd.py; the proposed patch adds configurable decoder units and selects float32 for the grown decoder. First inspect how Standard decoder shapes and checkpoint keys are loaded, then verify the new checkpoint has no missing or unexpected keys and matches the reference decode. The issue also raises unresolved TF32 and tiled decoding behavior, which need maintainer guidance.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Hello, and thank you for native YuE2 (#16250).

We are Ruach Studio (https://github.com/igrbible/Ruach_Studio, AGPL-3.0), an open local studio built on YuE2-3B.

We published YuE2-VAE-RS x3 v1 and v2: m-a-p's Standard decoder grown to 200.6 M parameters (three residual units at the latent rate, three more in each upsampling block, Standard's own operators), trained on music; it decodes the same latents. Weights CC BY-NC 4.0, float32 only; code Apache-2.0: https://huggingface.co/goldhub/YuE2-RS-Finetunes/tree/main/VAE-RS

What stock ComfyUI does with it today, in the YuE2 branch of comfy/sd.py: AudioOobleckVAE is built with Standard's shape and filled with load_state_dict(..., strict=False), so a checkpoint carrying x3 under vae.decoder.* loses the 210 tensors of the new units without an error. Measured: 23 % of the signal's RMS away from the real x3. And working_dtypes puts float16 first, while x3 is float32 only (guide §5).

What we offer: VAE-RS/kit/comfyui-native.patch (against master of November 2026, 29 lines added, 6 changed). DecoderBlock(extra_units=0) and OobleckDecoder(pre_units=0, extra_units=0) in comfy/ldm/audio/autoencoder.py, zero by default so Standard is untouched; in comfy/sd.py the YuE2 branch counts decoder.pre.<i> and the units of decoder.layers.1 and pins working_dtypes = [torch.float32] for the grown decoder. Tested: no missing or unexpected key, and a decode bit for bit equal to our reference.

Two points we would like your view on before a PR: TF32 (cuDNN convolutions run TF32 by default on Ampere and newer, which misses our verify.py tolerance of 1e-4 tenfold), and tiling (VAEDecodeAudioTiled blends a 64-frame overlap, while x3 needs 60 frames of cropped context each side to equal the whole decode; Standard needs 12).

Integration guide: https://huggingface.co/goldhub/YuE2-RS-Finetunes/blob/main/docs/VAE-RS-x3-integration.md
Study: https://ruachstudio.art/#vae-study · announcement: https://github.com/igrbible/Ruach_Studio/discussions/6

We are glad to open the PR (rebased on #16955 if it lands first, since both touch comfy/sd.py) and to answer questions.

Viktor Zhuromskyy and Claude (Anthropic), Ruach Studio

Lingua principale
Python
Stelle
146k
Fork
17.4k
Merge medio
19h 44m
PR unite (30g)
178

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di Comfy-Org/ComfyUI

Tutte le issue di Comfy-Org/ComfyUI

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.