Research: SINQ / A-SINQ on Qwen3.8-27B with the Q3_K_XL recipe — results

Offen
#199 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen

Dieses Issue hat noch niemand übernommen.

Bewertung

Schwierigkeit
4/5
Geschätzter Aufwand
3-5 Tage
Anfängerfreundlichkeit
35/100
Issue-Typ
Dokumentation
Klarheit
Größtenteils klar
Aktivitätsstatus
Aktiv
Tech-Stack
cpp, python

Rechercherichtung

Start by reviewing the SINQ/A-SINQ measurements and practical notes in the issue, then inspect tools/sinq/export_activations.py for the reported MTP-layer and CPU Triton failures. The report documents local workarounds but does not include exact commands or calibration data; clarify the reproducible steps and validate any exporter findings before documenting them.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Beschreibung

Research Stage
  • Background Research (Let's try to avoid reinventing the wheel)
  • Hypothesis Formed (How do you think this will work and it's effect?)
  • Strategy / Implementation Forming
  • Analysis of results
  • Debrief / Documentation (So people in the future can learn from us)
Analysis

I tried the SINQ / A-SINQ implementation from the v0.5.0 tag (f0beb9f60, i.e. before the offline optimizer refactor, so the old --sinq-calibration path) on Qwen3.8-27B, reproducing unsloth's UD-Q3_K_XL recipe exactly (504 per-tensor types from the original GGUF, output Q5_K, token_embd Q3_K, unsloth's published imatrix). The plain control is byte-identical to upstream llama.cpp llama-quantize with the same arguments, so the baseline is clean.

Variants (all 13.15 GB, control 13.15 GB):

  • Control: imatrix only
  • SINQ: --sinq-precondition=sidecar, 468 tensors
  • A-SINQ 8k: calibration = 8 × 1024 tokens from my own coding-agent sessions (chat template, tool calls), 128 rows
  • A-SINQ BF16: calibration = 12 × 2048 tokens where the BF16 model wrote the answers itself (mixed prompts: chat, tools, agentic, reasoning, code review, German, multilingual), 256 rows

Mean KLD vs BF16 (with BOS forced at the beginning of every chunk¹, -ub 128):

Control SINQ A-SINQ 8k A-SINQ BF16 UD-Q3_K_XL
wikitext-2, 20 × 4096 0.0399 0.0402 0.0421 0.0377 0.0305
chat conversation, 7 × 2048 0.0615 0.0590 0.0611 0.0532 0.0513
django source, 20 × 4096 0.0600 0.0601 0.0619 0.0605 0.0630
code (coderppl), 5 × 16384 0.01026 0.01054 0.01051 0.01022 0.00976
agentic traces, 8 × 16384 0.0208 0.0209 0.0215 0.0209 0.0209
bartowski calibration-v6, 20 × 4096 0.168 0.184 0.185 0.130 0.120
agentic same top token 93.69 % 92.48 % 92.34 % 92.44 % 93.59 %

Observations:

  • SINQ on top of an imatrix is neutral to slightly worse here (within ±3 % on five texts, +9 % on calibration-v6).
  • A-SINQ depends entirely on the calibration data. With coding-session calibration it was equal or slightly worse everywhere. With self-generated BF16 text it improves prose and chat clearly vs the control (wikitext −6 %, chat −14 %, calibration-v6 −23 %) while code and agentic stay unchanged.
  • It does not close the gap to unsloth's own UD-Q3_K_XL, which is still better on wikitext (−19 %), calibration-v6 (−8 %), code (−4 %) and chat (−4 %).
  • All SINQ variants lose ~1.2 points of top-token agreement on the agentic traces at equal KLD. Control and UD-Q3_K_XL do not.
  • The local L1 gain does not predict the model-level result. For example, ssm_beta always picks a strong alpha with a ~56 % local L1 gain, but that is measured before the sigmoid, so it may not mean much.

¹ A warning about measuring quality with llama-perplexity on Qwen3.8-27B: the model has no BOS (add_bos_token false), so every perplexity window starts mid-document without any start token. In some windows even BF16 loses track, resulting in abnormal results. Forcing a start token (--override-kv tokenizer.ggml.add_bos_token=bool:true, which uses <|endoftext|>) fixes it, and without it the small method differences flip sign:

  • wikitext: without BOS, A-SINQ BF16 looks worst (+10 % vs control); with BOS it is the best of the four (−6 %).
  • calibration-v6: without BOS, A-SINQ BF16 is equal to the control; with BOS it is −23 %.
  • coderppl: without BOS, the mean KLD is 7–9× higher and the 99.9 % KLD ~25× higher, and A-SINQ 8k looks 25 % worse than the control; with BOS, all variants are within 3 %.
  • Unaffected: UD-Q3_K_XL stays ahead either way, and django and the agentic traces barely change.

This is model-specific: on Qwen3.8-Flash-Next (MoE) the same override slightly hurts (PPL +1 % code, +6 % calibration-v6), and a Llama-based model (Bielik 7B) doesn't care either way. So it's worth checking both variants per model before comparing quant methods by PPL/KLD.

Practical notes:

  • tools/sinq/export_activations.py with a GGUF that contains an MTP layer: block_count, layer types and the tensor check fail, because the GGUF has 65 layers vs 64 in HF. I worked around it in a local copy.
  • An installed flash-linear-attention makes transformers pick Triton kernels on CPU, which fails with "Pointer argument cannot be accessed from Triton". Blocking fla and causal_conv1d in the exporter fixes it.
  • Both exporter points are from f0beb9f60; from the source they look unchanged on the current v0.5.0 branch, but I haven't re-run it there.
  • All quants above were made with the unmodified SINQ/A-SINQ code at f0beb9f60 (my build carries only unrelated local patches).

If you need more details (exact commands, per-chunk numbers, the calibration data) or have any questions, I'm happy to help.

AI Disclosure: The analysis, measurements and this text were prepared with AI assistance (Claude Opus 5.5), reviewed and edited by me.

Vorherrschende Sprache
C++
Sterne
1.2k
Forks
80
Ø Merge
4 T. 20 Std.
Gemergte PRs (30 T.)
2

Entwicklungsumgebung

Dieses Projekt bietet weder Dev-Container noch Dockerfile noch Beitragsleitfaden – die Einrichtung liegt bei Ihnen. Beginnen Sie mit der README; die allgemeinen Schritte stehen in unserem Leitfaden für den ersten Beitrag.

Erste Schritte

  1. Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
  2. Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
  3. Forken Sie das Repository und arbeiten Sie in einem Branch.
  4. Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.

Mehr aus Anbeeld/beellama.cpp

Alle Issues in Anbeeld/beellama.cpp

Ähnliche Issues

Weitere Issues zu C++

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.