Research: SINQ / A-SINQ on Qwen3.8-27B with the Q3_K_XL recipe — results
Dieses Issue hat noch niemand übernommen.
Bewertung
- Schwierigkeit
- 4/5
- Geschätzter Aufwand
- 3-5 Tage
- Anfängerfreundlichkeit
- 35/100
- Issue-Typ
- Dokumentation
- Klarheit
- Größtenteils klar
- Aktivitätsstatus
- Aktiv
- Bereich
- machine-learning
Rechercherichtung
Start by reviewing the SINQ/A-SINQ measurements and practical notes in the issue, then inspect tools/sinq/export_activations.py for the reported MTP-layer and CPU Triton failures. The report documents local workarounds but does not include exact commands or calibration data; clarify the reproducible steps and validate any exporter findings before documenting them.
Vom Indexierungsmodell aus dem Issue-Text verfasst.
Beschreibung
Research Stage
- Background Research (Let's try to avoid reinventing the wheel)
- Hypothesis Formed (How do you think this will work and it's effect?)
- Strategy / Implementation Forming
- Analysis of results
- Debrief / Documentation (So people in the future can learn from us)
Analysis
I tried the SINQ / A-SINQ implementation from the v0.5.0 tag (f0beb9f60, i.e. before the offline optimizer refactor, so the old --sinq-calibration path) on Qwen3.8-27B, reproducing unsloth's UD-Q3_K_XL recipe exactly (504 per-tensor types from the original GGUF, output Q5_K, token_embd Q3_K, unsloth's published imatrix). The plain control is byte-identical to upstream llama.cpp llama-quantize with the same arguments, so the baseline is clean.
Variants (all 13.15 GB, control 13.15 GB):
- Control: imatrix only
- SINQ:
--sinq-precondition=sidecar, 468 tensors - A-SINQ 8k: calibration = 8 × 1024 tokens from my own coding-agent sessions (chat template, tool calls), 128 rows
- A-SINQ BF16: calibration = 12 × 2048 tokens where the BF16 model wrote the answers itself (mixed prompts: chat, tools, agentic, reasoning, code review, German, multilingual), 256 rows
Mean KLD vs BF16 (with BOS forced at the beginning of every chunk¹, -ub 128):
| Control | SINQ | A-SINQ 8k | A-SINQ BF16 | UD-Q3_K_XL | |
|---|---|---|---|---|---|
| wikitext-2, 20 × 4096 | 0.0399 | 0.0402 | 0.0421 | 0.0377 | 0.0305 |
| chat conversation, 7 × 2048 | 0.0615 | 0.0590 | 0.0611 | 0.0532 | 0.0513 |
| django source, 20 × 4096 | 0.0600 | 0.0601 | 0.0619 | 0.0605 | 0.0630 |
| code (coderppl), 5 × 16384 | 0.01026 | 0.01054 | 0.01051 | 0.01022 | 0.00976 |
| agentic traces, 8 × 16384 | 0.0208 | 0.0209 | 0.0215 | 0.0209 | 0.0209 |
| bartowski calibration-v6, 20 × 4096 | 0.168 | 0.184 | 0.185 | 0.130 | 0.120 |
| agentic same top token | 93.69 % | 92.48 % | 92.34 % | 92.44 % | 93.59 % |
Observations:
- SINQ on top of an imatrix is neutral to slightly worse here (within ±3 % on five texts, +9 % on calibration-v6).
- A-SINQ depends entirely on the calibration data. With coding-session calibration it was equal or slightly worse everywhere. With self-generated BF16 text it improves prose and chat clearly vs the control (wikitext −6 %, chat −14 %, calibration-v6 −23 %) while code and agentic stay unchanged.
- It does not close the gap to unsloth's own UD-Q3_K_XL, which is still better on wikitext (−19 %), calibration-v6 (−8 %), code (−4 %) and chat (−4 %).
- All SINQ variants lose ~1.2 points of top-token agreement on the agentic traces at equal KLD. Control and UD-Q3_K_XL do not.
- The local L1 gain does not predict the model-level result. For example,
ssm_betaalways picks a strong alpha with a ~56 % local L1 gain, but that is measured before the sigmoid, so it may not mean much.
¹ A warning about measuring quality with llama-perplexity on Qwen3.8-27B: the model has no BOS (add_bos_token false), so every perplexity window starts mid-document without any start token. In some windows even BF16 loses track, resulting in abnormal results. Forcing a start token (--override-kv tokenizer.ggml.add_bos_token=bool:true, which uses <|endoftext|>) fixes it, and without it the small method differences flip sign:
- wikitext: without BOS, A-SINQ BF16 looks worst (+10 % vs control); with BOS it is the best of the four (−6 %).
- calibration-v6: without BOS, A-SINQ BF16 is equal to the control; with BOS it is −23 %.
- coderppl: without BOS, the mean KLD is 7–9× higher and the 99.9 % KLD ~25× higher, and A-SINQ 8k looks 25 % worse than the control; with BOS, all variants are within 3 %.
- Unaffected: UD-Q3_K_XL stays ahead either way, and django and the agentic traces barely change.
This is model-specific: on Qwen3.8-Flash-Next (MoE) the same override slightly hurts (PPL +1 % code, +6 % calibration-v6), and a Llama-based model (Bielik 7B) doesn't care either way. So it's worth checking both variants per model before comparing quant methods by PPL/KLD.
Practical notes:
tools/sinq/export_activations.pywith a GGUF that contains an MTP layer:block_count, layer types and the tensor check fail, because the GGUF has 65 layers vs 64 in HF. I worked around it in a local copy.- An installed
flash-linear-attentionmakes transformers pick Triton kernels on CPU, which fails with "Pointer argument cannot be accessed from Triton". Blockingflaandcausal_conv1din the exporter fixes it. - Both exporter points are from
f0beb9f60; from the source they look unchanged on the current v0.5.0 branch, but I haven't re-run it there. - All quants above were made with the unmodified SINQ/A-SINQ code at
f0beb9f60(my build carries only unrelated local patches).
If you need more details (exact commands, per-chunk numbers, the calibration data) or have any questions, I'm happy to help.
AI Disclosure: The analysis, measurements and this text were prepared with AI assistance (Claude Opus 5.5), reviewed and edited by me.
- Vorherrschende Sprache
- C++
- Sterne
- 1.2k
- Forks
- 80
- Ø Merge
- 4 T. 20 Std.
- Gemergte PRs (30 T.)
- 2
Entwicklungsumgebung
Dieses Projekt bietet weder Dev-Container noch Dockerfile noch Beitragsleitfaden – die Einrichtung liegt bei Ihnen. Beginnen Sie mit der README; die allgemeinen Schritte stehen in unserem Leitfaden für den ersten Beitrag.
Erste Schritte
- Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
- Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
- Forken Sie das Repository und arbeiten Sie in einem Branch.
- Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.
Mehr aus Anbeeld/beellama.cpp
-
[BUG] KVarN assertion failure in build_inp_mem_hybrid_iswa() due to uninitialized self_kvarn_rotOffen
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 68/100
Anbeeld/beellama.cpp#205 ·
-
Schwierigkeit 3/5 Ein halber Tag Anfängerfreundlichkeit 58/100
Anbeeld/beellama.cpp#206 ·
-
Schwierigkeit 3/5 1-2 Tage Anfängerfreundlichkeit 57/100
Anbeeld/beellama.cpp#204 ·
-
Schwierigkeit 4/5 3-5 Tage Anfängerfreundlichkeit 45/100
Anbeeld/beellama.cpp#203 · 2 Kommentare ·
-
Schwierigkeit 4/5 3-5 Tage Anfängerfreundlichkeit 45/100
Anbeeld/beellama.cpp#202 · 1 Kommentar ·
Alle Issues in Anbeeld/beellama.cpp
Ähnliche Issues
-
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 72/100
CachyOS/packageinstaller#144 ·
Maintainer antworten meist innerhalb von 1 Tag
-
documentation
Schwierigkeit 1/5 Unter einer Stunde Anfängerfreundlichkeit 82/100
TomerAberbach/profiler-md#951 ·
Maintainer antworten meist innerhalb von 1 Tag
-
[Bug] Default key routing (JavaStringHash) picks a different partition than the Java client for about half of keysEvtl. vergeben @0z5a hat das heute übernommen. Offen
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 82/100
apache/pulsar-client-go#1575 ·
Maintainer antworten meist innerhalb von 2 Tagen
-
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 72/100
pocoproject/poco#5764 ·
Maintainer antworten meist innerhalb von 2 Tagen
-
bug cpp-core-guidelines low-priority
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 62/100
NatronGitHub/openfx-supportext#164 ·
Maintainer antworten meist innerhalb von 1 Tag