LTXV Tiled Fusion Sampler: support downscaled IC-LoRA guides so the Pixel Spatial Upscaler can reach 4K

Aperta
  • Qualcosa non va?
#582 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
55/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
python
Ambito
backend

Direzione di ricerca

Read _crop_guide_metadata in tiled_fusion_sampler.py (the issue points to lines 134–139) and trace how it crops keyframe_idxs using the recorded latent_shape. Compare that with the dilated guide latent and mask from LTXAddVideoICLoRAGuide and LTXVDilateLatent. Done when the tiled sampler accepts factor 2 guides and the Pixel Spatial Upscaler can be exercised on an HD-to-4K tiled workflow; the issue does not name a test file.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Hello,

We use LTX 2.5 in ComfyUI for a Blender extension, Platformless AI, and the Pixel Spatial Upscaler (Lightricks/LTX-2.5-22b-IC-LoRA-Pixel-Spatial-Upscaler, ltx-2.5-22b-ic-lora-pixel-spatial-upscaler-x2-1.0.safetensors) is excellent. At HD it does exactly what the card shows: 960x544 to 1920x1088 at 121 frames gives separate strands of fur and crisp whiskers, well beyond a conventional enlargement or the Refine Details IC-LoRA. What we most want from it is HD to 4K, and we've not been able to find a way to get there.

Untiled, a single SamplerCustomAdvanced pass from 1920x1088 to 3840x2176 at 49 frames runs out of GPU memory on the first step on a 96 GB RTX PRO 6000. The HD job already peaks close to the top of the card at 121 frames, so this is much as expected.

Tiling is ruled out before we begin, because the upscaler needs its guide at half the output size (LTXAddVideoICLoRAGuide with latent_downscale_factor 2), and the LTXV Tiled Fusion Sampler won't accept that:

Downscaled (factor > 1) IC-LoRA guides are not supported; re-attach the guide at factor 1.

The check sits in _crop_guide_metadata in tiled_fusion_sampler.py (lines 134 to 139 on master, file last changed in c75eaca; likewise in bf2ca02, which is what we run). The tiled sampler with the Refine Details IC-LoRA, as in your LTX-2.5_V2V_TiledFusion_Upscale workflow, is how we take HD to 4K at the moment, and it works well. We'd be glad if the Pixel Spatial Upscaler could run through it in the same way.

From reading the code it looks like a contained change, though we may well be missing something. LTXAddVideoICLoRAGuide already dilates the half-size guide latent onto the full canvas grid (LTXVDilateLatent puts the real cells at every second position and masks the rest), so the guide latent and its mask crop at canvas coordinates just as a factor 1 guide does. The difference appears to be the guide's block of keyframe_idxs: _crop_guide_metadata reshapes it by the recorded latent_shape, which is the grid before dilation, [F, H/2, W/2]. For a guide at factor k, a tile covering canvas rows y0 to y0 + th would take the guide rows from ceil(y0 / k) to ceil((y0 + th) / k), and the same for columns. The positions would still be shifted by the tile's pixel origin as they are now, since keyframe_idxs hold canvas pixel positions, and latent_shape and pre_filter_count would be set to suit the cropped guide. Whether the upscaler behaves as well on tiles as on whole frames is for you to judge; the sampler's design of normalising positions to the tile suggests it might.

We haven't patched it ourselves, as we only run your released nodes. Could the tiled sampler support guides at latent_downscale_factor 2, so that the Pixel Spatial Upscaler can take 1920x1088 to 3840x2176? We'll test it on HD to 4K as soon as it's available.

Thank you for the models and the nodes.

Jules
Platformless AI

Lingua principale
Python
Stelle
4.2k
Fork
508
Merge medio
7h 14m
PR unite (30g)
3

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di Lightricks/ComfyUI-LTXVideo

Tutte le issue di Lightricks/ComfyUI-LTXVideo

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.