Research: SINQ / A-SINQ on Qwen3.8-27B with the Q3_K_XL recipe — results

未关闭
#199 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

还没有人认领这个 Issue。

评估

难度
4/5
预计耗时
3-5 天
新手友好度
35/100
Issue 类型
文档
描述清晰度
基本清楚
活跃度
活跃
技术栈
cpp, python

调研方向

Start by reviewing the SINQ/A-SINQ measurements and practical notes in the issue, then inspect tools/sinq/export_activations.py for the reported MTP-layer and CPU Triton failures. The report documents local workarounds but does not include exact commands or calibration data; clarify the reproducible steps and validate any exporter findings before documenting them.

由索引模型根据 Issue 内容生成。

描述

Research Stage
  • Background Research (Let's try to avoid reinventing the wheel)
  • Hypothesis Formed (How do you think this will work and it's effect?)
  • Strategy / Implementation Forming
  • Analysis of results
  • Debrief / Documentation (So people in the future can learn from us)
Analysis

I tried the SINQ / A-SINQ implementation from the v0.5.0 tag (f0beb9f60, i.e. before the offline optimizer refactor, so the old --sinq-calibration path) on Qwen3.8-27B, reproducing unsloth's UD-Q3_K_XL recipe exactly (504 per-tensor types from the original GGUF, output Q5_K, token_embd Q3_K, unsloth's published imatrix). The plain control is byte-identical to upstream llama.cpp llama-quantize with the same arguments, so the baseline is clean.

Variants (all 13.15 GB, control 13.15 GB):

  • Control: imatrix only
  • SINQ: --sinq-precondition=sidecar, 468 tensors
  • A-SINQ 8k: calibration = 8 × 1024 tokens from my own coding-agent sessions (chat template, tool calls), 128 rows
  • A-SINQ BF16: calibration = 12 × 2048 tokens where the BF16 model wrote the answers itself (mixed prompts: chat, tools, agentic, reasoning, code review, German, multilingual), 256 rows

Mean KLD vs BF16 (with BOS forced at the beginning of every chunk¹, -ub 128):

Control SINQ A-SINQ 8k A-SINQ BF16 UD-Q3_K_XL
wikitext-2, 20 × 4096 0.0399 0.0402 0.0421 0.0377 0.0305
chat conversation, 7 × 2048 0.0615 0.0590 0.0611 0.0532 0.0513
django source, 20 × 4096 0.0600 0.0601 0.0619 0.0605 0.0630
code (coderppl), 5 × 16384 0.01026 0.01054 0.01051 0.01022 0.00976
agentic traces, 8 × 16384 0.0208 0.0209 0.0215 0.0209 0.0209
bartowski calibration-v6, 20 × 4096 0.168 0.184 0.185 0.130 0.120
agentic same top token 93.69 % 92.48 % 92.34 % 92.44 % 93.59 %

Observations:

  • SINQ on top of an imatrix is neutral to slightly worse here (within ±3 % on five texts, +9 % on calibration-v6).
  • A-SINQ depends entirely on the calibration data. With coding-session calibration it was equal or slightly worse everywhere. With self-generated BF16 text it improves prose and chat clearly vs the control (wikitext −6 %, chat −14 %, calibration-v6 −23 %) while code and agentic stay unchanged.
  • It does not close the gap to unsloth's own UD-Q3_K_XL, which is still better on wikitext (−19 %), calibration-v6 (−8 %), code (−4 %) and chat (−4 %).
  • All SINQ variants lose ~1.2 points of top-token agreement on the agentic traces at equal KLD. Control and UD-Q3_K_XL do not.
  • The local L1 gain does not predict the model-level result. For example, ssm_beta always picks a strong alpha with a ~56 % local L1 gain, but that is measured before the sigmoid, so it may not mean much.

¹ A warning about measuring quality with llama-perplexity on Qwen3.8-27B: the model has no BOS (add_bos_token false), so every perplexity window starts mid-document without any start token. In some windows even BF16 loses track, resulting in abnormal results. Forcing a start token (--override-kv tokenizer.ggml.add_bos_token=bool:true, which uses <|endoftext|>) fixes it, and without it the small method differences flip sign:

  • wikitext: without BOS, A-SINQ BF16 looks worst (+10 % vs control); with BOS it is the best of the four (−6 %).
  • calibration-v6: without BOS, A-SINQ BF16 is equal to the control; with BOS it is −23 %.
  • coderppl: without BOS, the mean KLD is 7–9× higher and the 99.9 % KLD ~25× higher, and A-SINQ 8k looks 25 % worse than the control; with BOS, all variants are within 3 %.
  • Unaffected: UD-Q3_K_XL stays ahead either way, and django and the agentic traces barely change.

This is model-specific: on Qwen3.8-Flash-Next (MoE) the same override slightly hurts (PPL +1 % code, +6 % calibration-v6), and a Llama-based model (Bielik 7B) doesn't care either way. So it's worth checking both variants per model before comparing quant methods by PPL/KLD.

Practical notes:

  • tools/sinq/export_activations.py with a GGUF that contains an MTP layer: block_count, layer types and the tensor check fail, because the GGUF has 65 layers vs 64 in HF. I worked around it in a local copy.
  • An installed flash-linear-attention makes transformers pick Triton kernels on CPU, which fails with "Pointer argument cannot be accessed from Triton". Blocking fla and causal_conv1d in the exporter fixes it.
  • Both exporter points are from f0beb9f60; from the source they look unchanged on the current v0.5.0 branch, but I haven't re-run it there.
  • All quants above were made with the unmodified SINQ/A-SINQ code at f0beb9f60 (my build carries only unrelated local patches).

If you need more details (exact commands, per-chunk numbers, the calibration data) or have any questions, I'm happy to help.

AI Disclosure: The analysis, measurements and this text were prepared with AI assistance (Claude Opus 5.5), reviewed and edited by me.

主要语言
C++
星标
1.2k
派生
80
平均合并
4 天 20 小时
30 天内合并 PR
2

环境准备

这个项目没有提供开发容器、Dockerfile 或贡献指南,环境需要你自己搭建:先看它的 README,通用步骤见我们的新手贡献指南。

从这里开始

  1. 先读完整个 Issue,再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库,在一个分支上完成修改。
  4. 提交 Pull Request,并在描述里引用这个 Issue 编号。

Anbeeld/beellama.cpp 的其他 Issue

查看 Anbeeld/beellama.cpp 的全部 Issue

相似的 Issue

更多 C++ Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。