Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

[Feature Request] [Web] Expose kvCacheQuantizationBits in the onnxruntime-web WebGPU EP options

Aperta Adatta ai principianti
  • Qualcosa non va?
#33,752 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 2 giorni

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
2/5
Tempo stimato
1-3 ore
Idoneità per principianti
66/100
Tipo di issue
Funzionalità
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
javascript, typescript
Ambito
api, machine-learning

Direzione di ricerca

Inizia da js/web/lib/wasm/session-options.ts ed esamina la mappatura delle opzioni WebGPU e le definizioni dei tipi correlate in js/. Aggiungi kvCacheQuantizationBits alle opzioni WebGPU tipizzate e passalo come le opzioni esistenti; controlla i test e la documentazione pertinenti sulle opzioni di sessione WebGPU. Il lavoro è completato quando i chiamanti possono impostare 0, 4 o 8 tramite la Web API e il valore raggiunge il WebGPU EP.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

ep:WebGPU platform:web quantization
Describe the feature request

Please add kvCacheQuantizationBits (0 / 4 / 8) to InferenceSession.WebGpuExecutionProviderOption in onnxruntime-web, so it gets passed through like enableMatmulFp32Accumulation and the buffer cache modes already are.

Everything below comes from reading the source at 8d85527a0 and at current main (74bda515e). I haven't run any of it yet.

Where things stand

  • The native WebGPU EP reads ep.webgpuexecutionprovider.kvCacheQuantizationBits in ParseEpConfig (core/providers/webgpu/webgpu_provider_factory.cc).
  • js/web/lib/wasm/session-options.ts only maps a fixed list of WebGPU options to EP options: enableGraphCapture, preferredLayout, forceCpuNodeNames, validationMode, enableMatmulFp32Accumulation and the *BufferCacheMode keys. kvCacheQuantizationBits isn't one of them, and it doesn't appear in the js/ type definitions.
  • A workaround looks possible: sessionOptions.extra entries go through _OrtAddSessionConfigEntry, and for EP-device builds WebGpuEpFactory::CreateIExecutionProvider builds the provider from session_options->value.config_options. So this would probably work: 
    extra: { 'ep.webgpuexecutionprovider.kvCacheQuantizationBits': '8' }
    But it isn't documented, and I'm not sure it works on the minimal/legacy path, where the provider factory is created when the EP is appended, which happens before extra is applied.

Small related nit: sliding-window GQA layers now use flash attention (#32462, which shipped in 1.31.0), so KV cache quantization works with local_window_size. The error text in contrib_ops/webgpu/bert/group_query_attention.cc still says "Check that smooth_softmax and local_window_size are not set." Only smooth_softmax (and the other flash-attention preconditions) still rule out the flash path:

const int flash_local_window_size = kv_empty ? -1 : local_window_size_;
bool will_use_flash_attention = false;
if (!use_smooth_softmax_) {

At 8d85527a0 the gate was still if (!use_smooth_softmax_ && (!use_sliding_window || kv_empty)). That meant a quantized cache on a sliding-window layer would hit this error as soon as total_sequence_length passed the window. That part is fixed on main.

Describe scenario use case

We run small LLMs fully in the browser (Transformers.js + onnxruntime-web, WebGPU EP) on phones. One target is iPhone Safari, where a page gets about 2 GB in total, so memory, not compute, limits how long the context can be.

Our current model, LFM3-1.7B, is a hybrid: conv layers, full-attention GQA layers and sliding-window GQA layers (local_window_size 256), with 4 KV heads and head size 128. At 8K tokens its attention KV state is the biggest GPU cost after the weights, roughly 370 MB once both the past and present copies are counted for a step. Quantizing the KV cache to 8 or 4 bits would cut that by about 2x or 4x, which is enough to fit noticeably longer contexts on these devices.

A typed, documented JS option would make this usable from onnxruntime-web and from libraries built on it, like Transformers.js, which can only forward options that the JS API accepts.

Lingua principale
C++
Stelle
22.2k
Fork
4.4k
Merge medio
3g 6m
PR unite (30g)
302

Preparare l'ambiente

Apri in Codespaces

Avvia il container di sviluppo del progetto nel browser, con il tuo account GitHub.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di microsoft/onnxruntime

Tutte le issue di microsoft/onnxruntime

Issue simili

Altre issue su C++

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.