Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

iree-android: stateful prefill/step contract with a reusable prompt-prefix KV snapshot (and an embeddings-input variant for Vulkan)

Aperta
  • Qualcosa non va?
#410 3 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
30/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Ferma
Stack tecnologico
android, kotlin, machine-learning
Ambito
backend-api-design, mobile, performance

Direzione di ricerca

Start by reading the Android runtime contract and the FunctionGemma export path mentioned in FunctionGemmaExportHarness.kt:258–383; compare the exported gemma_prefill and gemma_with_past graphs with the runtime’s current stateless step(IntArray) path. The proposal also spans device-side KV lifetime, JNI input and error handling, Vulkan embeddings, and head slicing. Done means implementing and validating the stateful prefill, snapshot, step, and release contract, including the embeddings-input variant, against the supplied parity golden sets on the target device.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Motivation, with numbers

runtime-iree-android only offers the stateless redecode contract (step(IntArray) -> IntArray, whole sequence recomputed). On an arm32 Android device (32-bit process, 4× ARMv8, Mali) with FunctionGemma-270M FP32 and IREE 3.11.0:

SEQ CPU arm32 per step Vulkan per step (head sliced, embeddings gathered on the host)
64 10.4 s (6.6 s with the head sliced) 3.16 s
256 41.9 s 7.76 s
1024 77.3 s (sliced head) 29.9 s

A 15-token tool call against an 853-token catalog prompt therefore costs ≈ 19 min on the CPU and ≈ 7.5 min on the GPU. Prompt shortening does not fix it (short descriptions: −29 % tokens, −2/8 accuracy). With a KV contract that prefills the catalog once per process, snapshots the KV state, and per turn prefills only the utterance (≈ 36 tokens) plus ~15 decode steps, the same call projects to 8–29 s on the GPU. There is no other route to an interactive NLU on this class of device, and no arm64 firmware option (the product stays 32-bit).

Proposal

  1. Contract: prefill(ids) -> (lastId, kvHandle), snapshot(kvHandle) -> kvHandle', step(kvHandle, id) -> id, release(kvHandle); KV kept device-side (HAL buffers), never round-tripped through the JVM. The FunctionGemma export already emits gemma_prefill/gemma_with_past graphs (FunctionGemmaExportHarness.kt:258–383) — this is the runtime side of those.
  2. Head sliced to the target position in every graph (separate issue; 36 % of the step and the 32-bit memory cap).
  3. Embeddings-input variant (prefill(ids, embeddings)): IREE 3.11.0's SPIR-V backend cannot legalize the token-embedding gather (failed to legalize operation 'vector.step' on the gather dispatch). Gathering the 640-float rows on the host and passing tensor<1xSEQx640xf32> makes the same graph compile and run on Mali at 2.4–2.6× the CPU speed. The JNI currently accepts only IntArray.
  4. Error reporting through JNI (see the null-return issue).

I can provide the device, the measured graphs (MLIR with the sliced head and the host-gather input), the iree-benchmark-module scripts, and the golden sets for parity.

Context. Measured on 2026-09-03 while bringing FunctionGemma-270M up as an NLU cartridge on an arm32 Android device (Android 14, armeabi-v7a-only 32-bit process, 4× ARMv8 @ 2.0 GHz, Mali GPU) with SKaiNET 0.53.0 + SKaiNET-transformers 0.53.0 from Maven Central. Harness and raw result files: a local harness (can share on request).

Lingua principale
Kotlin
Stelle
8
Fork
3
Merge medio
12h 43m
PR unite (30g)
39

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di SKaiNET-developers/SKaiNET-transformers

Tutte le issue di SKaiNET-developers/SKaiNET-transformers

Issue simili

Altre issue su Kotlin

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.