Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Turn on speculative decoding for Gemma 4

Open
  • Is something wrong?
#218 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
Half a day
Newbie friendliness
55/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
dart, flutter
Domain
machine-learning, mobile-dev

Research direction

First check the Gemma 4 E2B load log and compare the generation chunks/sec with speculative decoding on and off on a real Android phone; the issue says to do this before coding. Then read lib/data/datasources/llm/flutter_gemma_gateway.dart, lib/data/datasources/llm/local_model_params.dart, and the AIModelRunner and lib/features/shiv/gana/engine/gana_workmanager.dart entry points. Done means speculative decoding is enabled for Gemma 4 only, tests cover other models remaining off or unset, and the measured result supports enabling it; this depends on #216.

Written by the indexing model from the issue text.

Description

enhancement P2-Medium

What

Turn on speculative decoding for the Gemma 4 E2B and E4B models.

What is speculative decoding (simple version)

A small, fast "draft" model guesses the next few words, and the big model checks them in one step. When the guesses are right, you get several words for the price of one, so answers come out faster.

Why

Google's model cards show a big speedup on an S26 Ultra GPU:

Model Without With
Gemma 4 E2B 51.5 tokens/sec 66–92 tokens/sec
Gemma 4 E4B 21.9 tokens/sec 37–49 tokens/sec

Things to know first

  • It speeds up writing the answer, not reading the prompt. Short outputs (Nataraj cards, translations) will gain less than the numbers above.
  • Only Gemma 4 supports it. Not Qwen3, not DeepSeek R1.
  • Today we never pass the setting, so the plugin uses whatever the model file defaults to. It may already be on, or off — we don't know yet.
  • Model files downloaded before May 5, 2026 don't have the draft model. They just ignore the setting — no crash, just no speedup. So no special handling is needed.

Step 1 — measure (do this first)

  1. Load Gemma 4 E2B on a real Android phone and check the log. The engine prints enable_speculative_decoding: true or false when it loads. If it's already true, we may be done.
  2. Run the same prompt with it on and off and compare the generation total ... chunks/sec log line.

Step 2 — code (if it helps)

  • Add an enableSpeculativeDecoding option to FlutterGemmaGateway.getActiveModel (lib/data/datasources/llm/flutter_gemma_gateway.dart)
  • Turn it on for Gemma 4 only (a flag in lib/data/datasources/llm/local_model_params.dart)
  • Pass it in AIModelRunner and in the background Gana path, which calls FlutterGemma.getActiveModel directly (lib/features/shiv/gana/engine/gana_workmanager.dart)
  • Tests for "on for Gemma 4, off/unset for other models"

Estimate: about half a day — 2–3 hours of code, ~1 hour measuring on a phone (plus the ~2.6 GB download if Gemma 4 isn't installed).

Depends on #216 (flutter_gemma 1.8.3 + GPU on Android).

Dominant language
Dart
Stars
2
Forks
2
Avg merge
1d 35m
Merged PRs (30d)
36

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from basictech01/uniun

All issues in basictech01/uniun

Similar issues

More Dart issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.