Turn on speculative decoding for Gemma 4
维护者通常 1 天内回复
还没有人认领这个 Issue。
评估
- 难度
- 3/5
- 预计耗时
- 半天
- 新手友好度
- 55/100
- Issue 类型
- 功能
- 描述清晰度
- 基本清楚
- 活跃度
- 活跃
调研方向
First check the Gemma 4 E2B load log and compare the generation chunks/sec with speculative decoding on and off on a real Android phone; the issue says to do this before coding. Then read lib/data/datasources/llm/flutter_gemma_gateway.dart, lib/data/datasources/llm/local_model_params.dart, and the AIModelRunner and lib/features/shiv/gana/engine/gana_workmanager.dart entry points. Done means speculative decoding is enabled for Gemma 4 only, tests cover other models remaining off or unset, and the measured result supports enabling it; this depends on #216.
由索引模型根据 Issue 内容生成。
描述
What
Turn on speculative decoding for the Gemma 4 E2B and E4B models.
What is speculative decoding (simple version)
A small, fast "draft" model guesses the next few words, and the big model checks them in one step. When the guesses are right, you get several words for the price of one, so answers come out faster.
Why
Google's model cards show a big speedup on an S26 Ultra GPU:
| Model | Without | With |
|---|---|---|
| Gemma 4 E2B | 51.5 tokens/sec | 66–92 tokens/sec |
| Gemma 4 E4B | 21.9 tokens/sec | 37–49 tokens/sec |
Things to know first
- It speeds up writing the answer, not reading the prompt. Short outputs (Nataraj cards, translations) will gain less than the numbers above.
- Only Gemma 4 supports it. Not Qwen3, not DeepSeek R1.
- Today we never pass the setting, so the plugin uses whatever the model file defaults to. It may already be on, or off — we don't know yet.
- Model files downloaded before May 5, 2026 don't have the draft model. They just ignore the setting — no crash, just no speedup. So no special handling is needed.
Step 1 — measure (do this first)
- Load Gemma 4 E2B on a real Android phone and check the log. The engine prints
enable_speculative_decoding: trueorfalsewhen it loads. If it's alreadytrue, we may be done. - Run the same prompt with it on and off and compare the
generation total ... chunks/seclog line.
Step 2 — code (if it helps)
- Add an
enableSpeculativeDecodingoption toFlutterGemmaGateway.getActiveModel(lib/data/datasources/llm/flutter_gemma_gateway.dart) - Turn it on for Gemma 4 only (a flag in
lib/data/datasources/llm/local_model_params.dart) - Pass it in
AIModelRunnerand in the background Gana path, which callsFlutterGemma.getActiveModeldirectly (lib/features/shiv/gana/engine/gana_workmanager.dart) - Tests for "on for Gemma 4, off/unset for other models"
Estimate: about half a day — 2–3 hours of code, ~1 hour measuring on a phone (plus the ~2.6 GB download if Gemma 4 isn't installed).
Depends on #216 (flutter_gemma 1.8.3 + GPU on Android).
- 主要语言
- Dart
- 星标
- 2
- 派生
- 2
- 平均合并
- 18 小时 53 分钟
- 30 天内合并 PR
- 35
环境准备
- 没有 Dockerfile 或 Docker Compose 文件
- 没有 Pull Request 模板
- 阅读贡献指南
从这里开始
- 先读完整个 Issue，再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库，在一个分支上完成修改。
- 提交 Pull Request，并在描述里引用这个 Issue 编号。
basictech01/uniun 的其他 Issue
-
bug good first issue
难度 2/5 1-3 小时 新手友好度 82/100
basictech01/uniun#300 ·
维护者通常 1 天内回复
-
bug good first issue
难度 3/5 1-2 天 新手友好度 52/100
basictech01/uniun#301 ·
维护者通常 1 天内回复
-
enhancement good first issue P2-Medium
难度 5/5 一周以上 新手友好度 25/100
basictech01/uniun#296 ·
维护者通常 1 天内回复
-
bug good first issue P2-Medium
难度 4/5 1-2 天 新手友好度 22/100
basictech01/uniun#293 ·
维护者通常 1 天内回复
-
bug P1-High
难度 4/5 3-5 天 新手友好度 18/100
basictech01/uniun#286 · 2 条评论 ·
维护者通常 1 天内回复
查看 basictech01/uniun 的全部 Issue
相似的 Issue
-
难度 2/5 1-3 小时 新手友好度 74/100
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 67/100
PhoebeSoftware/xrooster#41 ·
-
难度 2/5 1-3 小时 新手友好度 83/100
VeryGoodOpenSource/very_good_templates#665 ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 82/100
VeryGoodOpenSource/very_good_cli#1783 ·
维护者通常 1 天内回复
-
bug to check
难度 2/5 1-3 小时 新手友好度 66/100
维护者通常 1 天内回复