[Feature Request] Add DiCache for sample-adaptive caching in FLUX transformers
Maintainer antworten meist innerhalb von 1 Tag
Dieses Issue hat noch niemand übernommen.
Bewertung
- Schwierigkeit
- 5/5
- Geschätzter Aufwand
- Über eine Woche
- Anfängerfreundlichkeit
- 35/100
- Issue-Typ
- Feature
- Klarheit
- Größtenteils klar
- Aktivitätsstatus
- Aktiv
- Bereich
- backend, machine-learning
Rechercherichtung
Start by reading Diffusers’ existing caching API, cache-hook and state-management architecture, then inspect the FLUX transformer integration and its cache tests. The issue proposes adding a compatible DiCacheConfig with sample-adaptive probe decisions and trajectory-aligned residual reuse. Done means a FLUX-first integration without pipeline-specific monkey-patching, with regression tests and documentation; the issue provides no file paths or test names.
Vom Indexierungsmodell aus dem Issue-Text verfasst.
Beschreibung
Diffusers already supports caching techniques such as FirstBlockCache, but additional speed–quality trade-offs may be possible through sample-adaptive cache decisions and trajectory-aligned residual reuse.
I would like to propose integrating [DiCache (ICLR 2026)](https://proceedings.iclr.cc/paper_files/paper/2026/hash/78288ef33b18a351c3cd679dc9a15c8d-Abstract-Conference.html), which uses shallow transformer probes to determine whether a full forward pass is necessary and applies trajectory-aligned reuse when skipping computation.
Describe the solution you'd like.
Add a DiCacheConfig compatible with Diffusers' existing caching API:
from diffusers import DiCacheConfig
pipe.transformer.enable_cache(
DiCacheConfig(threshold=0.2, probe_depth=1)
)
# Generate images
pipe.transformer.disable_cache()
This is a proposed API, not an existing implementation. I propose an initial FLUX integration using the existing cache-hook and state-management architecture, without pipeline-specific monkey-patching.
Describe alternatives you've considered.
I benchmarked DiCache against Diffusers' FirstBlockCache and two DiCache ablations on pretrained FLUX.1-dev.
| Method | Speedup | SSIM ↑ | LPIPS ↓ |
|---|---|---|---|
| FirstBlockCache (0.05) | 1.23× | 0.9851 | 0.0107 |
| DiCache (0.2) | 2.14× | 0.9419 | 0.0515 |
| DiCache (0.3) | 2.42× | 0.9102 | 0.0834 |
| FirstBlockCache (0.2) | 2.69× | 0.8779 | 0.1301 |
| Fixed-schedule DiCache | 2.12× | 0.9348 | 0.0557 |
| Last-residual-only DiCache | 2.13× | 0.9274 | 0.0686 |
These results suggest that DiCache provides an additional operating region in the speed–fidelity trade-off. They do not establish superiority over a fully threshold-tuned FirstBlockCache Pareto frontier.
Additional context.
Experimental setup:
- Model:
aniketppanchal/flux.1-dev-nf4-pkg, revisiondde39613d8f3d7abe59d7c59ff16838b14bcebf8 - Hardware: Tesla T4
- Resolution: 512×512, 30 denoising steps
- Evaluation: 10 prompts × 3 seeds × 7 variants, with 2 timing repetitions each
- Versions: Diffusers 0.35.1, Transformers 4.56.1
- Measurements: denoising-only runtime, SSIM, PSNR, LPIPS, CLIP alignment, latent error, and GPU memory
Validation results:
All 210 method–prompt–seed configurations completed with zero recorded failures. Cache-disabled inference achieved exact tensor parity with native FLUX inference. Deterministic cache-reset and interleaved-generation tests also passed.
Per-timestep instrumentation demonstrated sample-dependent scheduling: DiCache produced 6 distinct cache schedules at threshold 0.2 and 3 at threshold 0.3 across 30 trajectories, compared with 1 schedule for the fixed-schedule control.
Ablations indicated improvements from both adaptive decisions and residual-history reuse.
Limitations: The experiments use an NF4-quantized checkpoint, not the paper's original full-precision setup. Fidelity can vary significantly between samples, with minimum SSIM values of 0.8120 and 0.7362 for DiCache thresholds 0.2 and 0.3, respectively. Additional matched-fidelity comparisons and broader model coverage would be valuable.
References:
- Paper: https://proceedings.iclr.cc/paper_files/paper/2026/hash/78288ef33b18a351c3cd679dc9a15c8d-Abstract-Conference.html
- Official implementation: https://github.com/Bujiazi/DiCache
I have reproducible benchmark scripts, paired images, cache trajectories, and validation records available.
I can prepare a minimal FLUX-first integration with the existing Diffusers caching infrastructure, together with regression tests and documentation. I would be happy to align the implementation with the maintainers' preferred design.
Would the maintainers be interested in this contribution? Any guidance on the preferred integration approach would be greatly appreciated.
If approved, I am willing to implement this feature and submit a pull request. I would also request this issue be assigned to me if possible.
- Vorherrschende Sprache
- Python
- Sterne
- 34.9k
- Forks
- 7.5k
- Ø Merge
- 3 T. 16 Std.
- Gemergte PRs (30 T.)
- 65
Entwicklungsumgebung
- Kein Dockerfile und keine Docker-Compose-Datei
- Hat eine Pull-Request-Vorlage
- Beitragsleitfaden lesen
Erste Schritte
- Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
- Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
- Forken Sie das Repository und arbeiten Sie in einem Branch.
- Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.
Mehr aus huggingface/diffusers
-
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 84/100
huggingface/diffusers#15363 ·
Maintainer antworten meist innerhalb von 1 Tag
-
`ZImageImg2ImgPipeline` / `ZImageInpaintPipeline`: explicit height/width different from the input image raises RuntimeErrorEvtl. vergeben @chessing234 hat das vor 1 Tag übernommen. Offenbug
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 70/100
huggingface/diffusers#15337 · 2 Kommentare ·
Maintainer antworten meist innerhalb von 1 Tag
-
Multi-vector textual inversion uses only the first vector when the token has uppercase letters (transformers v5)Evtl. vergeben @sunhouy hat das vor 2 Tagen übernommen. Offen
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 75/100
huggingface/diffusers#15313 ·
Maintainer antworten meist innerhalb von 1 Tag
-
TCDScheduler advances the generator on the final step of a strength-truncated custom scheduleEvtl. vergeben @SIDDARTHAREDDY8 hat das vor 5 Tagen übernommen. Offenbug
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 74/100
huggingface/diffusers#15288 · 2 Kommentare ·
Maintainer antworten meist innerhalb von 1 Tag
-
unload_textual_inversion(token) leaves the extra tokens of a multi-vector embedding behind, so it can't be loaded againEvtl. vergeben @sohailshk hat das vor 11 Tagen übernommen. Offen
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 72/100
huggingface/diffusers#15211 · 3 Kommentare ·
Maintainer antworten meist innerhalb von 1 Tag
Alle Issues in huggingface/diffusers
Ähnliche Issues
-
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 73/100
Maintainer antworten meist innerhalb von 1 Tag
-
Unused loot bag group names left in dreamcraft en_US.langEvtl. vergeben Ein verknüpfter Pull Request ist offen oder bereits gemergt. OffenStatus: Triage Suggestion
Schwierigkeit 1/5 Unter einer Stunde Anfängerfreundlichkeit 90/100
GTNewHorizons/GT-New-Horizons-Modpack#27789 ·
Maintainer antworten meist innerhalb von 1 Tag
-
bug frontend sev-5
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 83/100
-
core good first issue
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 65/100
-
b4b code health next
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 68/100
ESCOMP/CTSM#4330 · 1 Kommentar ·
Maintainer antworten meist innerhalb von 1 Tag