[Feature] Add support for Linum-AI/pjit-21k (Pyramid-JiT 21K text-to-image)

Open
  • Is something wrong?
#2,075 2 comments 0 reactions 1 assignee View on GitHub

Maintainers usually reply within 2 days

@leo2ultimate is already working on this.

Since Nov 12, 2026.

  • #2097 by @leo2ultimate — open

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Feature
Clarity
Clearly specified
Activity status
Active
Tech stack
python, pytorch
Domain
backend, machine-learning

Research direction

Start with the existing qwen_image21 pipeline and fastvideo/models/encoders/qwen3.py, then read the reference release’s generate.py to understand the model and sampler. The issue names fastvideo/registry.py, fastvideo/pipelines/basic/, fastvideo/configs/models/, and the basic inference and SSIM tests as integration points. Done means fixed-seed output matches the reference, with registry and SSIM coverage added.

Written by the indexing model from the issue text.

Description

enhancement help wanted installation performance scope: attention scope: docs scope: inference scope: kernel scope: model type: new-model

Model: Linum-AI/pjit-21k ("Pyramid-JiT 21K"), a text-to-image diffusion transformer released 2026-11-03.
Reference code: Linum-AI/pjit-21k · notes: linum.ai/field-notes/pjit-21k · license: Apache-2.0 (code and weights).

Why it fits FastVideo. FastVideo already ships T2I families (flux, flux2, sd35, qwen_image21), and this is a small DiT: 2.18B parameters, trained on 610M samples at 512px and 50M at 1024px. DiT weights are 8.7 GB fp32 and the text encoder about 8 GB bf16, so a single 24 GB card is plausible (the release does not quote peak memory). It is image-only: there is no video path in the release.

Port surface (from the reference release).

  • DiT: PyramidJiT, 2.18B, 3-D RoPE on image tokens, with height/width positions scaled x0.5 at 1024px (position interpolation from the 512px stage). Closest existing pipeline is qwen_image21.
  • Text conditioning: frozen Qwen3.5-4B, hidden layers 7, 15 and 27 concatenated through an MLP to 256 tokens. fastvideo/models/encoders/qwen3.py (Qwen3, used by Flux2 Klein) and qwen_image21.py (Qwen3-VL) are the nearest existing encoders; the 4B layer taps likely need a dedicated arch config.
  • Sampler: 50 Euler steps from noise (init scale 4), adaptive projected guidance at scale 15 against an empty negative prompt, bf16 autocast over fp32 state. The reference CLI exposes --prompt, --aspect_ratio, --seeds, --no_upsample.
  • Attention: varlen attention with pluggable backends plus a Triton noise kernel in the reference; this should map onto the attention backend registry.

Suggested first PR.

  1. Register Linum-AI/pjit-21k in fastvideo/registry.py; add a fastvideo/pipelines/basic/pjit/ pipeline (stages + presets) and DiT/encoder configs under fastvideo/configs/models/.
  2. Add examples/inference/basic/basic_pjit_21k.py plus a registry test, and SSIM coverage with reference videos stored alongside the existing SSIM tests.
  3. Check parity against the reference generate.py at a fixed seed before adding the model to the support matrix.

Effort: medium — a new DiT and encoder config plus a sampler stage, no new kernels if the existing attention backend covers varlen.

No open issue or PR mentions pjit or linum today, so this is the tracking issue. Comment here if you would like to take it.

— fv-superagent · run by @aryan5v

Dominant language
Python
Stars
4.6k
Forks
486
Avg merge
3d 15h
Merged PRs (30d)
74

Getting set up

  • No Dockerfile or Docker Compose file
  • Has a pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from hao-ai-lab/FastVideo

All issues in hao-ai-lab/FastVideo

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.