Research: SINQ / A-SINQ on Qwen3.8-27B with the Q3_K_XL recipe — results

オープン
#199 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

まだ誰も着手していません。

評価

難易度
4/5
見積もり時間
3〜5日
初心者へのやさしさ
35/100
issue の種類
ドキュメント
明瞭さ
おおむね明確
活発さ
活発
技術スタック
cpp, python

調査の方向性

Start by reviewing the SINQ/A-SINQ measurements and practical notes in the issue, then inspect tools/sinq/export_activations.py for the reported MTP-layer and CPU Triton failures. The report documents local workarounds but does not include exact commands or calibration data; clarify the reproducible steps and validate any exporter findings before documenting them.

索引モデルが issue の本文から書いたものです。

説明

Research Stage
  • Background Research (Let's try to avoid reinventing the wheel)
  • Hypothesis Formed (How do you think this will work and it's effect?)
  • Strategy / Implementation Forming
  • Analysis of results
  • Debrief / Documentation (So people in the future can learn from us)
Analysis

I tried the SINQ / A-SINQ implementation from the v0.5.0 tag (f0beb9f60, i.e. before the offline optimizer refactor, so the old --sinq-calibration path) on Qwen3.8-27B, reproducing unsloth's UD-Q3_K_XL recipe exactly (504 per-tensor types from the original GGUF, output Q5_K, token_embd Q3_K, unsloth's published imatrix). The plain control is byte-identical to upstream llama.cpp llama-quantize with the same arguments, so the baseline is clean.

Variants (all 13.15 GB, control 13.15 GB):

  • Control: imatrix only
  • SINQ: --sinq-precondition=sidecar, 468 tensors
  • A-SINQ 8k: calibration = 8 × 1024 tokens from my own coding-agent sessions (chat template, tool calls), 128 rows
  • A-SINQ BF16: calibration = 12 × 2048 tokens where the BF16 model wrote the answers itself (mixed prompts: chat, tools, agentic, reasoning, code review, German, multilingual), 256 rows

Mean KLD vs BF16 (with BOS forced at the beginning of every chunk¹, -ub 128):

Control SINQ A-SINQ 8k A-SINQ BF16 UD-Q3_K_XL
wikitext-2, 20 × 4096 0.0399 0.0402 0.0421 0.0377 0.0305
chat conversation, 7 × 2048 0.0615 0.0590 0.0611 0.0532 0.0513
django source, 20 × 4096 0.0600 0.0601 0.0619 0.0605 0.0630
code (coderppl), 5 × 16384 0.01026 0.01054 0.01051 0.01022 0.00976
agentic traces, 8 × 16384 0.0208 0.0209 0.0215 0.0209 0.0209
bartowski calibration-v6, 20 × 4096 0.168 0.184 0.185 0.130 0.120
agentic same top token 93.69 % 92.48 % 92.34 % 92.44 % 93.59 %

Observations:

  • SINQ on top of an imatrix is neutral to slightly worse here (within ±3 % on five texts, +9 % on calibration-v6).
  • A-SINQ depends entirely on the calibration data. With coding-session calibration it was equal or slightly worse everywhere. With self-generated BF16 text it improves prose and chat clearly vs the control (wikitext −6 %, chat −14 %, calibration-v6 −23 %) while code and agentic stay unchanged.
  • It does not close the gap to unsloth's own UD-Q3_K_XL, which is still better on wikitext (−19 %), calibration-v6 (−8 %), code (−4 %) and chat (−4 %).
  • All SINQ variants lose ~1.2 points of top-token agreement on the agentic traces at equal KLD. Control and UD-Q3_K_XL do not.
  • The local L1 gain does not predict the model-level result. For example, ssm_beta always picks a strong alpha with a ~56 % local L1 gain, but that is measured before the sigmoid, so it may not mean much.

¹ A warning about measuring quality with llama-perplexity on Qwen3.8-27B: the model has no BOS (add_bos_token false), so every perplexity window starts mid-document without any start token. In some windows even BF16 loses track, resulting in abnormal results. Forcing a start token (--override-kv tokenizer.ggml.add_bos_token=bool:true, which uses <|endoftext|>) fixes it, and without it the small method differences flip sign:

  • wikitext: without BOS, A-SINQ BF16 looks worst (+10 % vs control); with BOS it is the best of the four (−6 %).
  • calibration-v6: without BOS, A-SINQ BF16 is equal to the control; with BOS it is −23 %.
  • coderppl: without BOS, the mean KLD is 7–9× higher and the 99.9 % KLD ~25× higher, and A-SINQ 8k looks 25 % worse than the control; with BOS, all variants are within 3 %.
  • Unaffected: UD-Q3_K_XL stays ahead either way, and django and the agentic traces barely change.

This is model-specific: on Qwen3.8-Flash-Next (MoE) the same override slightly hurts (PPL +1 % code, +6 % calibration-v6), and a Llama-based model (Bielik 7B) doesn't care either way. So it's worth checking both variants per model before comparing quant methods by PPL/KLD.

Practical notes:

  • tools/sinq/export_activations.py with a GGUF that contains an MTP layer: block_count, layer types and the tensor check fail, because the GGUF has 65 layers vs 64 in HF. I worked around it in a local copy.
  • An installed flash-linear-attention makes transformers pick Triton kernels on CPU, which fails with "Pointer argument cannot be accessed from Triton". Blocking fla and causal_conv1d in the exporter fixes it.
  • Both exporter points are from f0beb9f60; from the source they look unchanged on the current v0.5.0 branch, but I haven't re-run it there.
  • All quants above were made with the unmodified SINQ/A-SINQ code at f0beb9f60 (my build carries only unrelated local patches).

If you need more details (exact commands, per-chunk numbers, the calibration data) or have any questions, I'm happy to help.

AI Disclosure: The analysis, measurements and this text were prepared with AI assistance (Claude Opus 5.5), reviewed and edited by me.

主要言語
C++
スター
1.2k
フォーク
80
平均マージ
4日 20時間
マージ済み PR(30日)
2

環境構築

このプロジェクトには開発コンテナ、Dockerfile、コントリビューションガイドがありません。まず README を読み、一般的な手順ははじめてのコントリビューションガイドを参照してください。

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

Anbeeld/beellama.cpp のほかの issue

Anbeeld/beellama.cpp の issue をすべて見る

似ている issue

C++ の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。