Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

[bug]: GGUF working-memory estimate misses the copies of aggregated sidecar patches (LoKr, LoHA, Full, DoRA, merged LoRA)

Aperta
  • Qualcosa non va?
#9,966 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 2 giorni

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
52/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
python, pytorch
Ambito
backend, performance

Direzione di ricerca

Read custom_linear.py at autocast_linear_forward_sidecar_patches and custom_module_mixin.py at _aggregate_patch_parameters, then inspect lora_layer_base.py at get_parameters. Reproduce the reported transient estimate for a patched GGUF Linear and trace how patch aggregation affects its reservation. Done when the estimate accounts for the aggregation path or that path avoids unnecessary dequantization, with coverage for the patch types listed in the issue.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

bug

Summary

peak_dequant_transient_bytes sizes a GGUF Linear's per-forward transient as one dequantized copy, the kernel intermediates and the packed bytes. Sidecar patches that are not plain LoRALayers take the aggregation path, which dequantizes the weight a second time and builds full-size delta weights. Those copies are counted for no GGUF type, so a patched layer can need more device memory than its node reserves.

Follow-up from the #9862 review.

Where

  • custom_linear.py, autocast_linear_forward_sidecar_patches: LoRALayer and FluxControlLoRALayer run as low-rank matmuls; every other patch goes to unprocessed_patches_and_weights and is aggregated.
  • custom_module_mixin.py, _aggregate_patch_parameters: dequantizes the GGUF weight (param.to(device=device).get_dequantized_tensor(), SDNQ the same) and sums each patch's get_parameters.
  • lora_layer_base.py, get_parameters: lora_weight and lora_weight * (weight * scale) are alive together, two full-size tensors.
  • Affected patch types: LoKr, LoHA, Full, DoRA, IA3, Norm, and MergedLayerPatch (diffusers-format FLUX LoRAs on BFL's fused layers).

Measured

RTX 4090, torch 2.13+cu130. One 3072×12288 GGUF Linear wrapped by apply_custom_layers_to_model, bf16 compute, packed weight resident on the device, input (1, 64, 12288). Device transient of one forward from max_memory_allocated, after a warm-up, against peak_dequant_transient_bytes minus the resident packed bytes:

Weight Patch Measured Estimate
Q8_0 none 4.0 B/el 4.0 B/el
Q8_0 LoRA, rank 16 4.0 B/el 4.0 B/el
Q8_0 LoKr, full-size Kronecker product 6.0 B/el 4.0 B/el
IQ4_NL none 2.0 B/el 2.0 B/el
IQ4_NL LoKr 6.0 B/el 2.0 B/el

The IQ4_NL rows use #9862, where host-decoded types reserve no device intermediates on dedicated VRAM; before it, their 8 B/el worst case happened to cover the aggregation.

For a 4608×53248 layer that is about 0.5 GB short with Q8_0 and about 1 GB with host-decoded types. Only a single patch was measured. With several, the running sum stays alive next to the following patch's copies (from the code, not measured).

Impact

FLUX.2, Krea-2, Z-Image and MiniMax H3 add 0.5 GB per LoRA, sized for the low-rank activation branch; FLUX.1, Qwen-Image and LTX-2 add nothing. A GGUF model with a LoKr, LoHA, Full, DoRA or diffusers-format FLUX LoRA can therefore exceed its working-memory reservation on a tight card, ending in an OOM or a cache that offloaded too little. Not observed as an OOM in a full generation; derived from the code path and the single-layer measurement.

Possible directions

  • Count the aggregation in the transient when a module carries non-LoRA patches: one dequantized copy plus two full-size copies at the patch dtype.
  • LoKr, LoHA, Full and Norm never read the original weight's values in get_weight (get_parameters only needs its shape), so _aggregate_patch_parameters could hand them a meta tensor instead of dequantizing. DoRA and IA3 do read it.
Lingua principale
TypeScript
Stelle
28.8k
Fork
3.1k
Merge medio
3g 2h
PR unite (30g)
150

Preparare l'ambiente

  • Nessun Dockerfile né file Docker Compose
  • Ha un modello di pull request
  • Nessuna guida per i contributori

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di invoke-ai/InvokeAI

Tutte le issue di invoke-ai/InvokeAI

Issue simili

Altre issue su TypeScript

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.