perf(server): keep CUDA graph replay when LoRA adapters are loaded

Aperta
#840 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
20/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
python, pytorch

Direzione di ricerca

Start at packages/sie_server/src/sie_server/adapters/modernbert_flash/__init__.py and the guard if self._graphs is None or self._peft_model is not None: return None; compare the base and adapter request paths with the graph-capture code cited in #423/#721. Read draft #829 for the runtime-LoRA direction and #647 for Qwen3 batching. Done means base and adapted requests replay graphs on ModernBERT/ColBERT and GLiNER2.5-Decide, parity stays within 0.02 of eager, L4 p50 is reported, and adapter eviction frees recordings within the existing memory budget.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Problem

Loading a LoRA adapter takes a model off CUDA graph replay, and on short encoder calls the eager path is several times slower.

  • On main, the ModernBERT and ColBERT flash adapters run every request eagerly once any PEFT LoRA is loaded, base-model requests included (packages/sie_server/src/sie_server/adapters/modernbert_flash/__init__.py: if self._graphs is None or self._peft_model is not None: return None). For gte-modernbert-base on an L4, graphs are worth 17.5 → 2.4 ms per query (#423).
  • Draft #829 (runtime LoRA for GLiNER2.5-Decide, Laya, pplx-decider and GLiClass) keeps graphs for base requests, but adapted requests run eagerly. Eager Decide on an L4 is 46.8 ms p50 against 8.4 ms with graphs (#721), so a tuned Decide would be about 5x slower than the base it was tuned from.
  • Qwen3-Embedding-4B gives up its four batches in flight when LoRA is configured (#647).

Goal

Requests that select a LoRA adapter replay recorded graphs as base requests do, under the same numerics rule as the other graph work: no probability moves more than 0.02 against eager, or the result is no worse against float32.

Possible approaches

  1. Static adapter buffers. Capture graphs with the LoRA A/B matrices read from fixed buffers. Before replay, copy the selected adapter's weights into those buffers, so adapters swap without a recapture. Needs one graph set per rank and target-module layout.
  2. Per-adapter recordings for hot adapters. Record a graph set per (adapter, bucket) inside the existing graph memory budget, and evict a recording together with its adapter in the LoRA LRU.
  3. Merge at staging. Merge a hot adapter into a copy of the base weights and record graphs for that copy. Simplest numerically, but it costs memory per adapter.

Whichever approach is chosen, base-model graphs must stay unaffected when adapters are loaded.

Acceptance

  • Base requests keep graph replay while any adapter is loaded, starting with the ModernBERT and ColBERT flash adapters.
  • Adapted requests replay graphs on GLiNER2.5-Decide and at least one ModernBERT model. Report parity numbers and L4 p50 before and after.
  • Graph memory stays within the existing budget, and evicting an adapter frees its recordings.
Lingua principale
Python
Stelle
3.5k
Fork
328
Merge medio
16h 25m
PR unite (30g)
121

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di superlinked/sie

Tutte le issue di superlinked/sie

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.