Kandinsky 6: add context-parallel (_cp_plan) support to Kandinsky6Transformer3DModel
Les mainteneurs répondent en général sous 1 jour
Personne n'a encore pris cette issue.
Évaluation
- Difficulté
- 4/5
- Temps estimé
- 3-5 jours
- Accessibilité débutants
- 45/100
- Type d'issue
- Fonctionnalité
- Clarté
- Plutôt claire
- Activité
- Active
- Domaine
- machine-learning
Piste de recherche
Find the Kandinsky6Transformer3DModel class in the diffusers source and inspect how _cp_plan is defined for LTX-2 (a similar joint audio+video model) to use as template. Add a _cp_plan attribute covering the visual and audio token streams, then verify with enable_parallelism(ContextParallelConfig(...)) under torchrun. Done means context parallelism runs without errors on 2 GPUs at batch size 1.
Rédigé par le modèle d'indexation à partir du texte de l'issue.
Description
Is your feature request related to a problem? Please describe.
On 2× T4, Kandinsky6Transformer3DModel sharded with device_map="balanced" keeps only one GPU busy at a time during denoising. A 200 ms nvidia-smi trace shows the GPUs taking turns on every step; both are busy in only 3 % of samples, and combined utilization is about one GPU's worth. This is expected for layer-wise model parallelism at batch size 1, so a second GPU adds memory but does not reduce per-clip latency. Context parallelism would let both GPUs work on the same clip, but in diffusers 0.42.1 Kandinsky6Transformer3DModel defines no _cp_plan (nor _tp_plan), so enable_parallelism(config=ContextParallelConfig(...)) is not available for it.
Raw trace: nvidia-smi-denoise-1793367281356898082.csv (timestamp, GPU index, utilization.gpu %, memory.used MiB, power.draw W)
Setup:
- Checkpoint:
kandinskylab/Kandinsky-6.0-Lite-distill-5s-Diffusers@4c487e8b4aeeb9059e5a77905aa2ddb59ec28f81 - Hardware: Kaggle 2× Tesla T4 (16 GB each), DiT in fp16 (T4 has no native bf16)
- Versions:
diffusers==0.42.1,transformers==5.22.1,accelerate==1.13.0, torch2.11.0+cu128 - Sampling: T2V, 432×256, 25 frames, 10 steps (
PiflowScheduler,n_grid=10),guidance_scale=1.0, batch size 1, joint video+audio (sample_audio=True)
repo, rev = "kandinskylab/Kandinsky-6.0-Lite-distill-5s-Diffusers", "4c487e8b4aeeb9059e5a77905aa2ddb59ec28f81"
transformer = Kandinsky6Transformer3DModel.from_pretrained(
repo, subfolder="transformer", revision=rev, torch_dtype=torch.float16,
device_map="balanced", max_memory={0: "10GiB", 1: "10GiB"})
pipe = Kandinsky6TI2VAPipeline.from_pretrained(
repo, revision=rev, transformer=transformer, text_encoder=None, text_encoder_2=None,
tokenizer=None, tokenizer_2=None, torch_dtype={"default": torch.float32})
# No pipe.to(...) / enable_model_cpu_offload(), so the device map is kept.
# Prompt embeddings are precomputed in a separate text-encoding stage and passed in.
output = pipe(prompt_embeds=..., pooled_prompt_embeds=..., num_inference_steps=10,
guidance_scale=1.0, sample_audio=True, output_type="latent")
Measurements (trace started right before and stopped right after the pipe(...) call; 67 polls ≈ 13.4 s):
| cuda:0 | cuda:1 | |
|---|---|---|
Mean utilization.gpu |
45.4 % | 51.3 % |
| Share of polls ≥ 50 % | 41.8 % | 53.7 % |
Peak memory.used (incl. CUDA context and allocator cache) |
3.36 GiB | 3.43 GiB |
| Both GPUs ≥ 50 % | Exactly one | Neither |
|---|---|---|
| 3.0 % | 89.6 % | 7.5 % |
Each of the 10 steps (~1.3 s) runs through the cuda:0 blocks and then the cuda:1 blocks.
Describe the solution you'd like.
Add a _cp_plan to Kandinsky6Transformer3DModel so that batch-1 inference can use context parallelism:
from diffusers import ContextParallelConfig
pipe.transformer.enable_parallelism(config=ContextParallelConfig(ulysses_degree=2)) # under torchrun, 2 processes
In 0.42.1, _cp_plan is already defined for Wan, LTX, LTX-2, Flux, Flux2, QwenImage, Cosmos3, ChronoEdit, Helios and MiniMax-H3. LTX-2 is also a joint audio+video model, so its plan may be a good template for Kandinsky 6's dual-stream video/audio blocks. Open questions:
- Should only the visual token stream be sharded, or the audio stream as well (it is much shorter)?
- Do the text/visual cross-attention and the rotary embeddings need special handling in the plan?
Describe alternatives you've considered.
- Current setup:
device_map="balanced"on the transformer only. It works and fits memory, but GPUs alternate (data above), so per-clip latency does not improve. Is this the recommended multi-GPU setup forKandinsky6TI2VAPipelineuntil context parallelism exists, or is there a better-supported pipeline-level strategy? - One full pipeline per GPU. At this resolution each GPU uses only ~3.4 GiB, so the whole DiT (5.94 GiB fp16) should fit on one T4 (not tested yet). Two independent clips at once would double throughput, but not reduce per-clip latency. Fit at the full 121-frame 864×480 size is unknown.
- CFG parallelism. Not applicable: the distilled Lite model runs with
guidance_scale=1.0, so there is no cond/uncond pair to split. - Pipeline-level
device_map. It places whole components on devices; since stages run one after another, it does not make two GPUs compute at the same time.
Additional context.
- Notebook (Kaggle, 2× T4, version 4): https://www.kaggle.com/code/applehawk/kandinsky-6-lite-diffusers-generation
- Pipeline placement in that notebook: text encoders on
cuda:1(Qwen with partial CPU offload), DiT split bydevice_map="balanced", video VAE (fp32, tiled) oncuda:1, audio VAE + vocoder (fp32) oncuda:0; each stage is loaded and freed before the next. utilization.gpuis the share of time a kernel was running, not how much compute was used; the 50 % threshold is my choice, and the plot shows the alternation regardless of it.- Scope: a single run at the small Smoke resolution; the full Baseline size and a single-GPU comparison have not been measured yet.
- Langage dominant
- Python
- Étoiles
- 34.9k
- Forks
- 7.5k
- Merge moyen
- 3 j 16 h
- PR mergées (30 j)
- 65
Préparer son environnement
- Aucun Dockerfile ni fichier Docker Compose
- Propose un modèle de pull request
- Lire le guide de contribution
Par où commencer
- Lisez l'issue en entier, puis le guide de contribution du projet.
- Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
- Forkez le dépôt et travaillez sur une branche.
- Ouvrez une pull request qui référence le numéro de l'issue.
Autres issues de huggingface/diffusers
-
Difficulté 2/5 1-3 heures Accessibilité débutants 84/100
huggingface/diffusers#15363 ·
Les mainteneurs répondent en général sous 1 jour
-
`ZImageImg2ImgPipeline` / `ZImageInpaintPipeline`: explicit height/width different from the input image raises RuntimeErrorPeut-être pris @chessing234 l’a pris il y a 1 jour. Ouvertebug
Difficulté 2/5 1-3 heures Accessibilité débutants 70/100
huggingface/diffusers#15337 · 2 commentaires ·
Les mainteneurs répondent en général sous 1 jour
-
Multi-vector textual inversion uses only the first vector when the token has uppercase letters (transformers v5)Peut-être pris @sunhouy l’a pris il y a 2 jours. Ouverte
Difficulté 2/5 1-3 heures Accessibilité débutants 75/100
huggingface/diffusers#15313 ·
Les mainteneurs répondent en général sous 1 jour
-
TCDScheduler advances the generator on the final step of a strength-truncated custom schedulePeut-être pris @SIDDARTHAREDDY8 l’a pris il y a 5 jours. Ouvertebug
Difficulté 2/5 1-3 heures Accessibilité débutants 74/100
huggingface/diffusers#15288 · 2 commentaires ·
Les mainteneurs répondent en général sous 1 jour
-
unload_textual_inversion(token) leaves the extra tokens of a multi-vector embedding behind, so it can't be loaded againPeut-être pris @sohailshk l’a pris il y a 11 jours. Ouverte
Difficulté 2/5 1-3 heures Accessibilité débutants 72/100
huggingface/diffusers#15211 · 3 commentaires ·
Les mainteneurs répondent en général sous 1 jour
Toutes les issues de huggingface/diffusers
Issues similaires
-
Difficulté 2/5 1-3 heures Accessibilité débutants 73/100
Les mainteneurs répondent en général sous 1 jour
-
Unused loot bag group names left in dreamcraft en_US.langPeut-être pris Une pull request liée à cette issue est ouverte ou déjà fusionnée. OuverteStatus: Triage Suggestion
Difficulté 1/5 Moins d'une heure Accessibilité débutants 90/100
GTNewHorizons/GT-New-Horizons-Modpack#27789 ·
Les mainteneurs répondent en général sous 1 jour
-
bug frontend sev-5
Difficulté 2/5 1-3 heures Accessibilité débutants 83/100
-
core good first issue
Difficulté 2/5 1-3 heures Accessibilité débutants 65/100
-
b4b code health next
Difficulté 2/5 1-3 heures Accessibilité débutants 68/100
ESCOMP/CTSM#4330 · 1 commentaire ·
Les mainteneurs répondent en général sous 1 jour