Research: SINQ / A-SINQ on Qwen3.8-27B with the Q3_K_XL recipe — results

Đang mở
#199 0 bình luận 0 reaction 0 người được giao Xem trên GitHub

Chưa có ai nhận issue này.

Đánh giá

Độ khó
4/5
Thời gian dự kiến
3-5 ngày
Mức phù hợp với người mới
35/100
Loại issue
Tài liệu
Độ rõ ràng
Khá rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
cpp, python
Lĩnh vực
machine-learning

Hướng nghiên cứu

Start by reviewing the SINQ/A-SINQ measurements and practical notes in the issue, then inspect tools/sinq/export_activations.py for the reported MTP-layer and CPU Triton failures. The report documents local workarounds but does not include exact commands or calibration data; clarify the reproducible steps and validate any exporter findings before documenting them.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

Research Stage
  • Background Research (Let's try to avoid reinventing the wheel)
  • Hypothesis Formed (How do you think this will work and it's effect?)
  • Strategy / Implementation Forming
  • Analysis of results
  • Debrief / Documentation (So people in the future can learn from us)
Analysis

I tried the SINQ / A-SINQ implementation from the v0.5.0 tag (f0beb9f60, i.e. before the offline optimizer refactor, so the old --sinq-calibration path) on Qwen3.8-27B, reproducing unsloth's UD-Q3_K_XL recipe exactly (504 per-tensor types from the original GGUF, output Q5_K, token_embd Q3_K, unsloth's published imatrix). The plain control is byte-identical to upstream llama.cpp llama-quantize with the same arguments, so the baseline is clean.

Variants (all 13.15 GB, control 13.15 GB):

  • Control: imatrix only
  • SINQ: --sinq-precondition=sidecar, 468 tensors
  • A-SINQ 8k: calibration = 8 × 1024 tokens from my own coding-agent sessions (chat template, tool calls), 128 rows
  • A-SINQ BF16: calibration = 12 × 2048 tokens where the BF16 model wrote the answers itself (mixed prompts: chat, tools, agentic, reasoning, code review, German, multilingual), 256 rows

Mean KLD vs BF16 (with BOS forced at the beginning of every chunk¹, -ub 128):

Control SINQ A-SINQ 8k A-SINQ BF16 UD-Q3_K_XL
wikitext-2, 20 × 4096 0.0399 0.0402 0.0421 0.0377 0.0305
chat conversation, 7 × 2048 0.0615 0.0590 0.0611 0.0532 0.0513
django source, 20 × 4096 0.0600 0.0601 0.0619 0.0605 0.0630
code (coderppl), 5 × 16384 0.01026 0.01054 0.01051 0.01022 0.00976
agentic traces, 8 × 16384 0.0208 0.0209 0.0215 0.0209 0.0209
bartowski calibration-v6, 20 × 4096 0.168 0.184 0.185 0.130 0.120
agentic same top token 93.69 % 92.48 % 92.34 % 92.44 % 93.59 %

Observations:

  • SINQ on top of an imatrix is neutral to slightly worse here (within ±3 % on five texts, +9 % on calibration-v6).
  • A-SINQ depends entirely on the calibration data. With coding-session calibration it was equal or slightly worse everywhere. With self-generated BF16 text it improves prose and chat clearly vs the control (wikitext −6 %, chat −14 %, calibration-v6 −23 %) while code and agentic stay unchanged.
  • It does not close the gap to unsloth's own UD-Q3_K_XL, which is still better on wikitext (−19 %), calibration-v6 (−8 %), code (−4 %) and chat (−4 %).
  • All SINQ variants lose ~1.2 points of top-token agreement on the agentic traces at equal KLD. Control and UD-Q3_K_XL do not.
  • The local L1 gain does not predict the model-level result. For example, ssm_beta always picks a strong alpha with a ~56 % local L1 gain, but that is measured before the sigmoid, so it may not mean much.

¹ A warning about measuring quality with llama-perplexity on Qwen3.8-27B: the model has no BOS (add_bos_token false), so every perplexity window starts mid-document without any start token. In some windows even BF16 loses track, resulting in abnormal results. Forcing a start token (--override-kv tokenizer.ggml.add_bos_token=bool:true, which uses <|endoftext|>) fixes it, and without it the small method differences flip sign:

  • wikitext: without BOS, A-SINQ BF16 looks worst (+10 % vs control); with BOS it is the best of the four (−6 %).
  • calibration-v6: without BOS, A-SINQ BF16 is equal to the control; with BOS it is −23 %.
  • coderppl: without BOS, the mean KLD is 7–9× higher and the 99.9 % KLD ~25× higher, and A-SINQ 8k looks 25 % worse than the control; with BOS, all variants are within 3 %.
  • Unaffected: UD-Q3_K_XL stays ahead either way, and django and the agentic traces barely change.

This is model-specific: on Qwen3.8-Flash-Next (MoE) the same override slightly hurts (PPL +1 % code, +6 % calibration-v6), and a Llama-based model (Bielik 7B) doesn't care either way. So it's worth checking both variants per model before comparing quant methods by PPL/KLD.

Practical notes:

  • tools/sinq/export_activations.py with a GGUF that contains an MTP layer: block_count, layer types and the tensor check fail, because the GGUF has 65 layers vs 64 in HF. I worked around it in a local copy.
  • An installed flash-linear-attention makes transformers pick Triton kernels on CPU, which fails with "Pointer argument cannot be accessed from Triton". Blocking fla and causal_conv1d in the exporter fixes it.
  • Both exporter points are from f0beb9f60; from the source they look unchanged on the current v0.5.0 branch, but I haven't re-run it there.
  • All quants above were made with the unmodified SINQ/A-SINQ code at f0beb9f60 (my build carries only unrelated local patches).

If you need more details (exact commands, per-chunk numbers, the calibration data) or have any questions, I'm happy to help.

AI Disclosure: The analysis, measurements and this text were prepared with AI assistance (Claude Opus 5.5), reviewed and edited by me.

Ngôn ngữ chính
C++
Star
1.2k
Fork
80
Merge trung bình
4 ngày 20 giờ
Pull request đã merge (30 ngày)
2

Chuẩn bị môi trường

Dự án này không cung cấp dev container, Dockerfile hay hướng dẫn đóng góp, nên bạn cần tự thiết lập môi trường: hãy bắt đầu từ README và xem hướng dẫn đóng góp lần đầu của chúng tôi để biết các bước chung.

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của Anbeeld/beellama.cpp

Tất cả issue của Anbeeld/beellama.cpp

Issue tương tự

Thêm issue về C++

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.