Hacktoberfest 2026：メンテナが10月に向けて印を付けた、オープンで初心者向けの issue。 Hacktoberfest の issue を見る

Apertus xIELU formula / param-space mismatch — forward pass NaN's on real Apertus-8B Q4_K_S

オープン
  • 何か問題がありますか？
#103 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

まだ誰も着手していません。

評価

難易度
5/5
見積もり時間
1週間以上
初心者へのやさしさ
30/100
issue の種類
バグ
明瞭さ
明確に書かれている
活発さ
静か
技術スタック
kotlin
領域
backend, machine-learning

調査の方向性

Read ApertusXIELU.kt at line 27 and compare its activation formula with the Hugging Face Apertus modeling code; inspect the layer 0 parameters in the reference checkpoint and compare them with the GGUF values. The issue also points to ApertusWeightLoader.kt, the DSL wiring in apertusNetwork(), and ApertusRealGgufLoadingTest for a forward pass probe. Done means identifying the source of the NaNs and confirming the corrected activation produces finite logits.

索引モデルが issue の本文から書いたものです。

説明

Summary

After PR #102 the load → convert → fromWeights → forward chain runs to completion against unsloth/Apertus-8B-Instruct-2509-GGUF (Q4_K_S) — but the logit vector is all-NaN at the first forward pass. The propagation source is ApertusXIELU.xielu(buf, params) in layer 0.

Diagnosis

ApertusXIELU.kt evaluates:

alpha_p_eff = softplus(alpha_p)
alpha_n_eff = beta + softplus(alpha_n)

if x > 0:  alpha_p_eff * x*x + beta * x
else:      (expm1(min(x, eps)) - x) * alpha_n_eff + beta * x

with softplus(x) = if (x > 20) x else ln(1 + exp(x)).

The unsloth GGUF stores per-layer xIELU params (verified via gguf Python lib):

layer alpha_p alpha_n beta eps
0 166.0 40.75 0.5 -9.98e-7
1 174.0 31.625 0.5 -9.98e-7
2 128.0 22.875 0.5 -9.98e-7
… (decaying) (decaying) 0.5 -9.98e-7
11 -0.477 … 0.5 -9.98e-7
… (small / negative) … 0.5 -9.98e-7

For layer 0, softplus(166) ≈ 166 (the > 20 shortcut applies), so the activation collapses to 166·x² + 0.5·x. Any input value |x| past a few causes the FFN output to overflow, and the post-FFN matmul + residual chain NaN-poisons the rest of the forward.

Hypothesis

Either:

  1. The unsloth GGUF stores alpha_p / alpha_n in a different transform space than ApertusXIELU.kt expects. softplus(stored) may already be applied (i.e. these are the post-softplus alpha_p_eff values, not the raw learned parameters). In that case the right thing is to drop the softplus calls in the activation and use the stored values directly — which would still leave layer 0 with alpha_p_eff = 166, and the activation would still overflow for any non-trivial input.
  2. ApertusXIELU.kt's formula diverges from the Apertus-8B reference. Possible candidates:
    • The reference may scale alpha_p by some normalization factor.
    • The if x > 0 branch might use alpha_p_eff * (e^x - x - 1) (an ELU-like form) rather than alpha_p_eff * x².
    • beta, eps, etc. may participate in a different combination.
  3. The unsloth converter is upstream-buggy and writes the wrong values in the wrong fields.

What to verify

  • Pull the HF reference checkpoint swiss-ai/Apertus-8B-Instruct-2509 and inspect the actual act_fn parameter values for layer 0 — see whether they match the unsloth-stored values or are very different (e.g. small).
  • Diff ApertusXIELU.xielu against the activation forward in the HF Apertus modeling code (modeling_apertus.py or similar).
  • Quick sanity probe: replace ApertusXIELU with a no-op linear (or with silu) and re-run the forward pass. If logits become finite, the failure is fully inside xIELU.
  • If a different unsloth quant of Apertus is available, repro under it to rule out a unsloth-conversion glitch isolated to Q4_K_S.

Where the formula lives

  • llm-inference/apertus/src/commonMain/kotlin/sk/ainet/models/apertus/ApertusXIELU.kt:27
  • The DSL wiring is apertusNetwork() line 65: stage.xielu(id = \"act_fn\"). The DSL-emitted op is in skainet-lang-core; need to confirm it computes the same thing as ApertusXIELU.xielu reference.
  • Per-layer params are loaded by extractXIELUParams / extractXIELUParamsFromStreamingMeta in ApertusWeightLoader.kt, then converted to scalar tensors named blk.N.mlp.act_fn.{alpha_p,alpha_n,beta,eps} by ApertusNetworkLoader.applyWeightsToNetwork.

Out of scope

This issue is only about the activation. Loader / module construction / Q4_K dispatch are all correct (PR #102 verified loader + fromGguf().load() end-to-end on real Apertus-8B). When the activation is fixed, forward / generate / tool calling smoke tests in ApertusRealGgufLoadingTest should pass without further loader work.

主要言語
Kotlin
スター
8
フォーク
3
平均マージ
12時間 43分
マージ済み PR（30日）
39

環境構築

このプロジェクトには開発コンテナ、Dockerfile、コントリビューションガイドがありません。まず README を読み、一般的な手順ははじめてのコントリビューションガイドを参照してください。

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

SKaiNET-developers/SKaiNET-transformers のほかの issue

SKaiNET-developers/SKaiNET-transformers の issue をすべて見る

似ている issue

Kotlin の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。