Add Liger kernels + FLCE `token_accuracy` plumbing for Muse Glimmer (`muse_glimmer`)
Maintainers usually reply within 3 days
@buffett0323 is already working on this.
Since Aug 14, 2026.
Assessment
This issue has not been assessed yet.
Description
🚀 The feature, motivation and pitch
Feature
Add Muse Glimmer (model_type=muse_glimmer, e.g. MuseGlimmerForConditionalGeneration in recent transformers) to Liger’s supported model map, with
the same pattern used for other CausalLM / VLM families:
- FusedLinearCrossEntropy / skip-logits loss path for training (avoid materializing full
[B, S, V]logits). return_token_accuracy=True→outputs.token_accuracyon the model output (e.g. viaLigerForCausalLMLoss+LigerCausalLMOutputWithPast,
as in the Qwen3 patch), so trainers that request the metric can log it without a custom fallback.
Motivation
We train Muse Glimmer-30B with TRL SFTTrainer + use_liger_kernel=True (long context / FSDP2 + context parallel). TRL’s compute_loss does:
- set
inputs["return_token_accuracy"] = Truewhenuse_liger_kernel=True - skip the logits-based
mean_token_accuracypath - only read
outputs.token_accuracy
On Muse today: muse_glimmeris not inMODEL_TYPE_TO_APPLY_LIGER_FN, so_apply_liger_kernel_to_instanceis a no-op.- Stock Muse forward always runs
lm_head, returnsMuseGlimmerCausalLMOutputWithPastwithouttoken_accuracy, and sets
accepts_loss_kwargs=False. - Result: TRL warns “liger-kernel did not return token_accuracy when requested” and drops
mean_token_accuracyfrom training logs even though
loss/backprop still work.
Liger already implementsreturn_token_accuracyin fused CE ops and wires it for patched models (e.g. Qwen3). We want the same end-to-end path for
Muse so: - long-context SFT can use true fused CE (VRAM), and
- TRL/HF trainers can log token accuracy without maintaining a Muse-specific trainer subclass.
Related context: - TRL logging of Liger token accuracy: https://github.com/huggingface/trl/pull/4302
- TRL behavior when accuracy is missing / None: https://github.com/huggingface/trl/issues/4730
- Muse modeling lives in Hugging Face
transformers(MuseGlimmerForConditionalGeneration); loss is currentlyself.loss_function(logits, labels, vocab_size, **kwargs)after always computing logits (incl. softcap /output_multiplier).
Pitch
Supporting muse_glimmer would make use_liger_kernel=True actually meaningful for a newly popular multimodal/instruct stack, and align Muse with
the contract TRL already assumes for Liger-backed SFT (fused loss + optional token_accuracy for metrics).
Alternatives
- Trainer-side fallback (what we do now)
Subclass TRLSFTTrainerand temporarily disable the Liger metrics branch so accuracy is computed from Muse’s existinglogits. Restores logging
only; does not give Muse fused CE / skip-logits savings. - Set
use_liger_kernel=False
Avoids the TRL warning and restores logits-basedmean_token_accuracy, but forgoes any future Liger benefits and still materializes full logits
(painful at longmax_length/ large vocab). - Vendor patch Muse forward locally
Manually callLigerForCausalLMLoss(..., return_token_accuracy=...)and return a custom output withtoken_accuracy. Works, but duplicates what
Liger’s model-specific patches already do for other architectures and drifts from upstreamtransformers. - Upgrade TRL only
Newer TRL versions harden None-handling for missing accuracy, but they still cannot inventtoken_accuracyif the model forward never returns it.
Muse still needs Liger (or equivalent) plumbing.
Preferred upstream fix: first-class Muse support in Liger (RMSNorm / SwiGLU / RoPE as applicable + FLCE withreturn_token_accuracyon the
ConditionalGeneration forward), registered inMODEL_TYPE_TO_APPLY_LIGER_FN.
Additional context
Muse Glimmer is a newly released model from Meta and we request for support
- Dominant language
- Python
- Stars
- 6.7k
- Forks
- 656
- Avg merge
- 4d 12h
- Merged PRs (30d)
- 27
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from linkedin/Liger-Kernel
-
Difficulty 2/5 1-3 hours Newbie friendliness 77/100
linkedin/Liger-Kernel#1658 ·
Maintainers usually reply within 3 days
-
LigerPolyNorm defaults to in_place=True (docstring says False) and overwrites shared gradientsPossibly taken @nazek52 claimed this 1 day ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
linkedin/Liger-Kernel#1647 ·
Maintainers usually reply within 3 days
-
[Bug] Experimental embedding kernels overflow int32 row offsets (illegal memory access past 2^31 output elements)Possibly taken @0xDexe claimed this 8 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
linkedin/Liger-Kernel#1570 · 3 comments ·
Maintainers usually reply within 3 days
-
Difficulty 3/5 1-2 days Newbie friendliness 68/100
linkedin/Liger-Kernel#1661 ·
Maintainers usually reply within 3 days
-
Difficulty 4/5 3-5 days Newbie friendliness 55/100
linkedin/Liger-Kernel#1660 ·
Maintainers usually reply within 3 days
All issues in linkedin/Liger-Kernel
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
GoogleCloudPlatform/professional-services-data-validator#1899 ·
Maintainers usually reply within 4 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
mohanraj00/entelos#200 ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 68/100
-
sweep
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
bketelsen/truenas-mcp#77 ·
Maintainers usually reply within 1 day
-
Console writes URL credentials to the local log, though AGENTS.md says secrets never reach diskOpen
Difficulty 1/5 Under an hour Newbie friendliness 86/100
sandbox-quantum/switch#2163 ·
Maintainers usually reply within 1 day