Tracking: Whisper-class offline ASR on Android — engine-side P0
维护者通常 1 天内回复
还没有人认领这个 Issue。
评估
- 难度
- 5/5
- 预计耗时
- 一周以上
- 新手友好度
- 20/100
- Issue 类型
- 功能
- 描述清晰度
- 基本清楚
- 活跃度
- 活跃
调研方向
Start with the linked P0 issues #1275–#1281 to identify a bounded engine-side task; this tracking issue spans native kernels, scheduling, VAD, ONNX import, and numerical tooling. Use the stated Pixel-class whisper-tiny.en exit criterion and M2-A5 harness as the completion guide: the encoder must be at least 5× faster than the scalar path, matmuls must name their kernels, and Silero VAD must match ONNX Runtime within 1e-4 without an ORT dependency.
由索引模型根据 Issue 内容生成。
描述
Goal
Make SKaiNET the obvious engine for a fully offline, Whisper-class transcription app on Android. The reference case is an existing app on LiteRT (Whisper large-v3-turbo split encoder/decoder on OpenCL FP16, Parakeet TDT 0.6B, Silero VAD on ONNX Runtime, a 350M LLM for titles on LiteRT-LM): three native runtimes in one APK, hand-written mel/FFT/resampler/tokenizers/decode loops, and model-file surgery plus byte-patched runtime AARs to work around GPU numerics bugs. Its author's Pixel 7 Pro numbers are the bar:
| Path | 14 s memo | RTF |
|---|---|---|
| LiteRT CPU, XNNPACK, 2×3 threads | 82.2 s | 5.9 |
| LiteRT GPU, OpenCL FP16 | 34.4 s | 2.5 |
Target for this milestone: CPU-only SKaiNET Whisper large-v3-turbo at or below 82 s on the same class of device, token-identical to whisper.cpp, with the audio side and VAD served by SKaiNET libraries.
What 0.54.0 already provides
- JNI NEON backend (
skainet-backend-jni-cpu) for every GGML quant format, runtime dotprod dispatch, threaded packed matmuls, self-installing dispatch (#920 → 0.50–0.52). - Off-heap mapped weights on Android (
AndroidGguf.loader,WeightForm(residency = MAPPED)), memory plan and device fit check before load; a 1.0 GB Q4_K_M LLM decodes at 61–66 ms/step on a Pixel 8a. Lstmwith explicit state (#823);transformer-coreKV cache; tape → StableHLO export at billion-parameter scale (0.53.0).
Engine-side P0 (this repo)
- #1275 — dense FP32/FP16 matmul on the JNI tier (attention is scalar on Android today)
- #1276 — native fused LayerNorm / GELU / softmax / conv1d kernels
- #1277 —
Scheduleon Android verified and measured for SDPA - #1278 — native Silero VAD model on
Lstm, weights from the upstream ONNX file - #1279 — ONNX importer: LSTM, Gather, Unsqueeze/Squeeze, Cast, Where, If
- #1280 —
TensorDump+TensorComparefor divergence localisation - #1281 — FP16 simulation context + FP32-accumulating LayerNorm lowering rule
Out of this repo, tracked with the ASR projects
- Whisper v3 128-mel
MelConfigin the audio library (HF-exact constants). - Whisper large-v3-turbo model + self-attention KV cache + timestamp rules + long-form windowing in the Whisper project, on top of the current engine pin.
- Consumer docs: the artifact picker must list
skainet-backend-jni-cpu; theAndroidGgufKDoc still says packed tensors heap-stage (fixed in 0.50.0).
Follow-ups already identified (P1, separate issues when P0 lands)
AndroidGguf.loaderover anAssetFileDescriptorso anoCompressGGUF is mapped straight from the APK instead of copied out.- Windowed mapping to lift the 2 GB single-region cap.
- HF
tokenizer.jsonbyte-level BPE loader for Whisper vocab. - Parakeet TDT cartridge reusing
Lstmand the 128-mel config.
Exit criterion
whisper-tiny.en runs on a Pixel-class phone through the JNI tier with mapped weights and the M2-A5 harness names the kernel behind every matmul; the encoder is ≥ 5× faster than the 0.54.0 scalar path; Silero VAD matches ONNX Runtime within 1e-4 with no ORT dependency.
- 主要语言
- Kotlin
- 星标
- 52
- 派生
- 15
- 平均合并
- 1 天 15 小时
- 30 天内合并 PR
- 36
环境准备
- 没有 Dockerfile 或 Docker Compose 文件
- 没有 Pull Request 模板
- 阅读贡献指南
从这里开始
- 先读完整个 Issue，再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库，在一个分支上完成修改。
- 提交 Pull Request，并在描述里引用这个 Issue 编号。
SKaiNET-developers/SKaiNET 的其他 Issue
-
coding good first issue size:xs skill:kotlin-core sub-issue
难度 2/5 1-3 小时 新手友好度 88/100
SKaiNET-developers/SKaiNET#1323 ·
维护者通常 1 天内回复
-
coding good first issue platform size:xs skill:js sub-issue
难度 2/5 1-3 小时 新手友好度 72/100
SKaiNET-developers/SKaiNET#1232 ·
维护者通常 1 天内回复
-
skeep tracking
难度 5/5 一周以上 新手友好度 15/100
SKaiNET-developers/SKaiNET#1331 ·
维护者通常 1 天内回复
-
assessment size:s skill:review sub-issue
难度 4/5 1-2 天 新手友好度 18/100
SKaiNET-developers/SKaiNET#1330 ·
维护者通常 1 天内回复
-
documentation good first issue size:s skill:docs sub-issue
难度 3/5 1-2 天 新手友好度 78/100
SKaiNET-developers/SKaiNET#1329 ·
维护者通常 1 天内回复
查看 SKaiNET-developers/SKaiNET 的全部 Issue
相似的 Issue
-
难度 2/5 1-3 小时 新手友好度 83/100
Kotlin/kotlin-lsp#288 ·
-
needs-triage
难度 2/5 1-3 小时 新手友好度 76/100
chrisbanes/haze#1536 ·
维护者通常 1 天内回复
-
Cassiopeia drops the `präsentiert von` credit printed in the event body可能已有人在做 关联的 PR 仍在进行中或已合并。 未关闭afk-ok area:data-quality importer size:S
难度 2/5 1-3 小时 新手友好度 77/100
enorm-labs/event-junkie#3166 ·
维护者通常 1 天内回复
-
enhancement good first issue
难度 2/5 1-3 小时 新手友好度 84/100
rajpreetcodes/Skills_MCP_Server#25 ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 82/100
com-lihaoyi/mill#7776 ·
维护者通常 1 天内回复