Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Eval bug: Laguna repeats <assistant> on Metal for ubatches >= 32 tokens (ffn_moe_down input exceeds F16 range)

Open Beginner friendly
  • Is something wrong?
#30,896 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
78/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
cpp
Domain
backend, performance

Research direction

The issue identifies src/llama-graph.cpp, build_moe_ffn, and the condition that currently enables src1 rescaling for Mistral4 and Mellum. Read that condition and compare Laguna's behavior with those architectures; the report says adding Laguna locally makes default Metal output match CPU. Done means the ubatch 32 reproduction no longer repeats <assistant> and agrees with CPU output.

Written by the indexing model from the issue text.

Description

Name and Version
version: 0.7.0-dev (build 11669, commit 620d2225c)
built with AppleClang 21.0.0.21000334 for Darwin arm64

The build_moe_ffn code involved is unchanged on master as of 2026-10-20.

Operating systems

Mac (macOS 26.7.1)

GGML backends

Metal

Hardware

Apple M4 Max, 64 GB

Models

poolside/Laguna-XS-2.1-GGUF Q4_K_M (arch laguna)

Problem description & steps to reproduce

On Metal, Laguna-XS-2.1 generates only a repeating <assistant> token (id 23) when the prompt is evaluated in a ubatch of 32 or more tokens. A ubatch of 31 or fewer, --cpu-moe, and CPU-only runs all give correct output.

llama-completion -m Laguna-XS-2.1-Q4_K_M.gguf -no-cnv --temp 0 -n 16 -c 4096 \
  -p $'<system>You are a helpful, conversationally-fluent assistant. Answer questions clearly and concisely, and think step by step before replying.</system>\n<user>Hi</user>\n<assistant><think>'

The prompt is 42 tokens.

Run Output
default, -ub 32 <assistant><assistant><assistant>…
-ub 31 Okay, the user said "Hi". I need to respond appropriately. Let me
--cpu-moe correct (same as above)
-ngl 0 --device none correct

The 32-token threshold matches ggml_metal_op_mul_mat_id_use_mm (ne21 >= 32). At that size the mul_mm_id kernel stages src1 in half precision.

Since #29029 (0f8a414b7), the src1 rescale in that kernel runs only when GGML_PREC_F32 is set on src1. build_moe_ffn sets it only for LLM_ARCH_MISTRAL4 || LLM_ARCH_MELLUM (#30584, the fix for the same symptom in #30579).

Laguna's ffn_moe_down input also exceeds F16 range. A -ngl sweep fits accumulated overflow across layers rather than a single bad layer: -ngl 2 is correct, while -ngl 3 and higher loop.

Fix, verified locally: in src/llama-graph.cpp build_moe_ffn, add || arch == LLM_ARCH_LAGUNA to that condition. With this change, Metal output at the default ubatch matches CPU.

First Bad Commit

Probably 0f8a414b7 (#29029), which made the rescale opt-in. Not bisected.

Relevant log output

No crash or error is logged; the output is wrong as shown in the table above.

Dominant language
C++
Stars
131k
Forks
24.4k
Avg merge
2d 1h
Merged PRs (30d)
504

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from ggml-org/llama.cpp

All issues in ggml-org/llama.cpp

Similar issues

More C++ issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.