Hacktoberfest 2026：メンテナが10月に向けて印を付けた、オープンで初心者向けの issue。 Hacktoberfest の issue を見る

Phase 4 readiness: validate DSL Qwen output against HF reference on real GGUFs

オープン
  • 何か問題がありますか？
#118 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

まだ誰も着手していません。

  • #119 @michalharakal による — マージされずにクローズ

評価

難易度
5/5
見積もり時間
1週間以上
初心者へのやさしさ
25/100
issue の種類
機能追加
明瞭さ
おおむね明確
活発さ
停滞
技術スタック
huggingface, kotlin
領域
machine-learning, testing-qa

調査の方向性

Start with tests/smoke/smoke-models.json and the existing tests/smoke/ harness to see how models are selected and run. Add a smoke comparison for the Qwen3-1.7B-Q8_0 GGUF on the DSL path, using a fixed prompt and temperature 0; choose either a checked-in Transformers reference fixture or a live llama.cpp reference. Done means the first N greedy tokens agree and top-K probability KL divergence is below the chosen threshold.

索引モデルが issue の本文から書いたものです。

説明

Replaces #114 with a focused scope. Phase-4 readiness gate before swapping the CLI Qwen branch from LlamaRuntime to OptimizedLLMRuntime + qwenNetwork().

Why this gate, not synthetic parity

The original #114 framed readiness as "DSL ↔ legacy LlamaRuntime numerical parity on identical synthetic weights". That framing has two problems:

  1. Legacy gets deleted in Phase 4. Once LlamaRuntime is gone, DSL-vs-legacy parity is permanently moot. Optimising for it is debt.
  2. Neither path is the source of truth. The real correctness reference is HuggingFace transformers (or llama.cpp) running the same Qwen3 GGUF — that's what users compare against. Synthetic-weight equivalence between two of our paths only tells us they're internally consistent, not that either is correct.

So the right gate is: DSL Qwen, fed a real Qwen3 GGUF, produces output that matches an HF reference within tolerance. The legacy path is allowed to disagree; if DSL matches HF and legacy doesn't, legacy was the wrong one all along.

Scope

Add an HF-reference smoke comparison to tests/smoke/ that:

  1. Takes a Qwen3-1.7B-Q8_0 GGUF (per existing tests/smoke/smoke-models.json), runs it through kllama-cli on the DSL path (after Phase 4) for a fixed prompt at temperature 0.
  2. Captures top-K next-token IDs / probabilities at each step.
  3. Compares against an external reference for the same prompt:
    • Option A: pre-computed reference logged via transformers Python script, checked into the repo as a fixture.
    • Option B: live comparison via llama.cpp CLI invocation (more setup, more runtime).

Pass criterion: top-1 token agrees with reference for the first N greedy steps (e.g. N=8); top-K probability KL-divergence below threshold.

Out of scope

  • DSL-vs-legacy LlamaRuntime numerical parity (the original #114 question). The fixes from #116 (RoPE NEOX) and #117 (QK-norm eps) shipped because they're independently correct, not because they close synthetic parity. Synthetic divergence remains (max |Δ| = 49.26 at pos 0 on the parity probe), root cause unknown — and that's fine, the ship gate now sits elsewhere.
  • Quantized parity. Phase 1B.2 (#113) already smoke-tested Q8 round-trip equivalence inside the DSL path. HF reference comparison can also be FP32-only initially.

Phase 4 — blocked on this gate

Phase 4 (CLI swap, delete LlamaRuntime family) should not merge until at least one greedy-token comparison passes against an HF reference on a real Qwen3 GGUF.

#52 was the other Phase-4 blocker; resolved by #115 (byte-level BPE).

Refs the closed #114, which has the historical investigation context.

主要言語
Kotlin
スター
8
フォーク
3
平均マージ
1日 16時間
マージ済み PR（30日）
48

環境構築

このプロジェクトには開発コンテナ、Dockerfile、コントリビューションガイドがありません。まず README を読み、一般的な手順ははじめてのコントリビューションガイドを参照してください。

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

SKaiNET-developers/SKaiNET-transformers のほかの issue

SKaiNET-developers/SKaiNET-transformers の issue をすべて見る

似ている issue

Kotlin の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。