Gemma4 text model support
Maintainers usually reply within 3 days
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 35/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Quiet
- Tech stack
- python
- Domain
- machine-learning
Research direction
Start with the existing apply_liger_kernel_to_gemma3_text and related Gemma patch entry points, then review how AutoLigerKernelForCausalLM and Hugging Face Trainer discover model types. Confirm the gemma4_text patch covers the listed RMSNorm, GeGLU, and CrossEntropyLoss/FLCE kernels while leaving RoPE unchanged; done means text-only Gemma 4 models are auto-patched without affecting multimodal support.
Written by the indexing model from the issue text.
Description
🚀 The feature, motivation and pitch
Add apply_liger_kernel_to_gemma4_text patching support for the Gemma 4 text model family.
Gemma 4 was released by Google DeepMind on April 2, 2026, as their most capable open model family to date. The series includes 31B (Dense), 26B-A4B (MoE), E4B, and E2B variants, all under Apache 2.0 license. These models are seeing rapid adoption for fine-tuning and downstream tasks.
Gemma 4's text backbone (gemma4_text in HuggingFace transformers) introduces several architectural changes from Gemma 3:
- Hybrid Sliding/Full Attention with GQA, alternating between sliding window (512 tokens) and full attention layers
- GeGLU activation (
gelu_pytorch_tanh) in the MLP, with conditional double-wide MLP for KV-shared layers - Standard RMSNorm — unlike Gemma 3 which uses
(1 + weight) * normed(zero-initialized weights with offset), Gemma 4 uses the standardweight * normed(one-initialized weights, no offset) - Per-layer v_norm with
with_scale=False(no learnable weight) in some attention heads - Large vocabulary (262,144 tokens), making FLCE particularly beneficial for memory savings
- RoPE with dual configurations:
theta=10000for sliding attention,theta=1000000with proportional scaling + partial rotary for full attention
Liger Kernel already supports the Gemma family:
Gemma— viaapply_liger_kernel_to_gemmaGemma2— viaapply_liger_kernel_to_gemma2Gemma3 (Text)— viaapply_liger_kernel_to_gemma3_textGemma3 (Multimodal)— viaapply_liger_kernel_to_gemma3
Gemma 4 registers as a separate model type (gemma4_text) in HuggingFace transformers (v5.5.0+), so a dedicated patch function is needed for auto-patching to work (e.g., via AutoLigerKernelForCausalLM or the HF Trainer's use_liger_kernel=True).
The following kernels are applicable:
| Kernel | Applicable? | Notes |
|---|---|---|
| RMSNorm | ✅ | Standard RMSNorm (offset=0.0), not the Gemma 3 offset variant. 6-7 instances per layer. |
| GeGLU | ✅ | Requires a custom wrapper (LigerGemma4TextMLP) due to (config, layer_idx) init signature and conditional double-wide MLP logic. |
| CrossEntropyLoss / FLCE | ✅ | Vocab size is 262,144 — FLCE memory savings are significant. |
| RoPE | ❌ | Gemma 4 uses apply_rotary_pos_emb(x, cos, sin) (single-tensor, called separately for q and k), incompatible with Liger's fused (q, k, cos, sin) variant. |
Models: gemma-4-31B · gemma-4-31B-it · gemma-4-26B-A4B · gemma-4-E4B · gemma-4-E2B
I have a working implementation and would be happy to open a PR.
Alternatives
The Gemma 3 text patch (apply_liger_kernel_to_gemma3_text) cannot be reused directly because:
- RMSNorm configuration differs (Gemma 3 uses
offset=1.0, init_fn="zeros"while Gemma 4 usesoffset=0.0, init_fn="ones") - The MLP class has a different
__init__signature ((config, layer_idx)with conditional double-wide logic) - The model type string is different (
gemma4_textvsgemma3_text), so auto-patching would not trigger
Additional context
This issue covers the text-only model (Gemma4ForCausalLM). Multimodal support (Gemma4ForConditionalGeneration) with vision/audio encoders could be addressed in a follow-up issue.
Alternatives
No response
Additional context
No response
- Dominant language
- Python
- Stars
- 6.7k
- Forks
- 656
- Avg merge
- 4d 12h
- Merged PRs (30d)
- 27
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from linkedin/Liger-Kernel
-
Difficulty 2/5 1-3 hours Newbie friendliness 77/100
linkedin/Liger-Kernel#1658 ·
Maintainers usually reply within 3 days
-
LigerPolyNorm defaults to in_place=True (docstring says False) and overwrites shared gradientsPossibly taken @nazek52 claimed this 1 day ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
linkedin/Liger-Kernel#1647 ·
Maintainers usually reply within 3 days
-
[Bug] Experimental embedding kernels overflow int32 row offsets (illegal memory access past 2^31 output elements)Possibly taken @0xDexe claimed this 8 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
linkedin/Liger-Kernel#1570 · 3 comments ·
Maintainers usually reply within 3 days
-
Difficulty 3/5 1-2 days Newbie friendliness 68/100
linkedin/Liger-Kernel#1661 ·
Maintainers usually reply within 3 days
-
Difficulty 4/5 3-5 days Newbie friendliness 55/100
linkedin/Liger-Kernel#1660 ·
Maintainers usually reply within 3 days
All issues in linkedin/Liger-Kernel
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
GoogleCloudPlatform/professional-services-data-validator#1899 ·
Maintainers usually reply within 4 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
mohanraj00/entelos#200 ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 68/100
-
sweep
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
bketelsen/truenas-mcp#77 ·
Maintainers usually reply within 1 day
-
Console writes URL credentials to the local log, though AGENTS.md says secrets never reach diskOpen
Difficulty 1/5 Under an hour Newbie friendliness 86/100
sandbox-quantum/switch#2163 ·
Maintainers usually reply within 1 day