Research: SINQ / A-SINQ on Qwen3.8-27B with the Q3_K_XL recipe — results
Personne n'a encore pris cette issue.
Évaluation
- Difficulté
- 4/5
- Temps estimé
- 3-5 jours
- Accessibilité débutants
- 35/100
- Type d'issue
- Documentation
- Clarté
- Plutôt claire
- Activité
- Active
- Domaine
- machine-learning
Piste de recherche
Start by reviewing the SINQ/A-SINQ measurements and practical notes in the issue, then inspect tools/sinq/export_activations.py for the reported MTP-layer and CPU Triton failures. The report documents local workarounds but does not include exact commands or calibration data; clarify the reproducible steps and validate any exporter findings before documenting them.
Rédigé par le modèle d'indexation à partir du texte de l'issue.
Description
Research Stage
- Background Research (Let's try to avoid reinventing the wheel)
- Hypothesis Formed (How do you think this will work and it's effect?)
- Strategy / Implementation Forming
- Analysis of results
- Debrief / Documentation (So people in the future can learn from us)
Analysis
I tried the SINQ / A-SINQ implementation from the v0.5.0 tag (f0beb9f60, i.e. before the offline optimizer refactor, so the old --sinq-calibration path) on Qwen3.8-27B, reproducing unsloth's UD-Q3_K_XL recipe exactly (504 per-tensor types from the original GGUF, output Q5_K, token_embd Q3_K, unsloth's published imatrix). The plain control is byte-identical to upstream llama.cpp llama-quantize with the same arguments, so the baseline is clean.
Variants (all 13.15 GB, control 13.15 GB):
- Control: imatrix only
- SINQ:
--sinq-precondition=sidecar, 468 tensors - A-SINQ 8k: calibration = 8 × 1024 tokens from my own coding-agent sessions (chat template, tool calls), 128 rows
- A-SINQ BF16: calibration = 12 × 2048 tokens where the BF16 model wrote the answers itself (mixed prompts: chat, tools, agentic, reasoning, code review, German, multilingual), 256 rows
Mean KLD vs BF16 (with BOS forced at the beginning of every chunk¹, -ub 128):
| Control | SINQ | A-SINQ 8k | A-SINQ BF16 | UD-Q3_K_XL | |
|---|---|---|---|---|---|
| wikitext-2, 20 × 4096 | 0.0399 | 0.0402 | 0.0421 | 0.0377 | 0.0305 |
| chat conversation, 7 × 2048 | 0.0615 | 0.0590 | 0.0611 | 0.0532 | 0.0513 |
| django source, 20 × 4096 | 0.0600 | 0.0601 | 0.0619 | 0.0605 | 0.0630 |
| code (coderppl), 5 × 16384 | 0.01026 | 0.01054 | 0.01051 | 0.01022 | 0.00976 |
| agentic traces, 8 × 16384 | 0.0208 | 0.0209 | 0.0215 | 0.0209 | 0.0209 |
| bartowski calibration-v6, 20 × 4096 | 0.168 | 0.184 | 0.185 | 0.130 | 0.120 |
| agentic same top token | 93.69 % | 92.48 % | 92.34 % | 92.44 % | 93.59 % |
Observations:
- SINQ on top of an imatrix is neutral to slightly worse here (within ±3 % on five texts, +9 % on calibration-v6).
- A-SINQ depends entirely on the calibration data. With coding-session calibration it was equal or slightly worse everywhere. With self-generated BF16 text it improves prose and chat clearly vs the control (wikitext −6 %, chat −14 %, calibration-v6 −23 %) while code and agentic stay unchanged.
- It does not close the gap to unsloth's own UD-Q3_K_XL, which is still better on wikitext (−19 %), calibration-v6 (−8 %), code (−4 %) and chat (−4 %).
- All SINQ variants lose ~1.2 points of top-token agreement on the agentic traces at equal KLD. Control and UD-Q3_K_XL do not.
- The local L1 gain does not predict the model-level result. For example,
ssm_betaalways picks a strong alpha with a ~56 % local L1 gain, but that is measured before the sigmoid, so it may not mean much.
¹ A warning about measuring quality with llama-perplexity on Qwen3.8-27B: the model has no BOS (add_bos_token false), so every perplexity window starts mid-document without any start token. In some windows even BF16 loses track, resulting in abnormal results. Forcing a start token (--override-kv tokenizer.ggml.add_bos_token=bool:true, which uses <|endoftext|>) fixes it, and without it the small method differences flip sign:
- wikitext: without BOS, A-SINQ BF16 looks worst (+10 % vs control); with BOS it is the best of the four (−6 %).
- calibration-v6: without BOS, A-SINQ BF16 is equal to the control; with BOS it is −23 %.
- coderppl: without BOS, the mean KLD is 7–9× higher and the 99.9 % KLD ~25× higher, and A-SINQ 8k looks 25 % worse than the control; with BOS, all variants are within 3 %.
- Unaffected: UD-Q3_K_XL stays ahead either way, and django and the agentic traces barely change.
This is model-specific: on Qwen3.8-Flash-Next (MoE) the same override slightly hurts (PPL +1 % code, +6 % calibration-v6), and a Llama-based model (Bielik 7B) doesn't care either way. So it's worth checking both variants per model before comparing quant methods by PPL/KLD.
Practical notes:
tools/sinq/export_activations.pywith a GGUF that contains an MTP layer:block_count, layer types and the tensor check fail, because the GGUF has 65 layers vs 64 in HF. I worked around it in a local copy.- An installed
flash-linear-attentionmakes transformers pick Triton kernels on CPU, which fails with "Pointer argument cannot be accessed from Triton". Blockingflaandcausal_conv1din the exporter fixes it. - Both exporter points are from
f0beb9f60; from the source they look unchanged on the current v0.5.0 branch, but I haven't re-run it there. - All quants above were made with the unmodified SINQ/A-SINQ code at
f0beb9f60(my build carries only unrelated local patches).
If you need more details (exact commands, per-chunk numbers, the calibration data) or have any questions, I'm happy to help.
AI Disclosure: The analysis, measurements and this text were prepared with AI assistance (Claude Opus 5.5), reviewed and edited by me.
- Langage dominant
- C++
- Étoiles
- 1.2k
- Forks
- 80
- Merge moyen
- 4 j 20 h
- PR mergées (30 j)
- 2
Préparer son environnement
Ce projet ne fournit ni conteneur de développement, ni Dockerfile, ni guide de contribution : l'installation est à votre charge. Commencez par son README, et consultez notre guide de la première contribution pour les étapes générales.
Par où commencer
- Lisez l'issue en entier, puis le guide de contribution du projet.
- Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
- Forkez le dépôt et travaillez sur une branche.
- Ouvrez une pull request qui référence le numéro de l'issue.
Autres issues de Anbeeld/beellama.cpp
-
[BUG] KVarN assertion failure in build_inp_mem_hybrid_iswa() due to uninitialized self_kvarn_rotOuverte
Difficulté 2/5 1-3 heures Accessibilité débutants 68/100
Anbeeld/beellama.cpp#205 ·
-
Difficulté 3/5 Une demi-journée Accessibilité débutants 58/100
Anbeeld/beellama.cpp#206 ·
-
Difficulté 3/5 1-2 jours Accessibilité débutants 57/100
Anbeeld/beellama.cpp#204 ·
-
Difficulté 4/5 3-5 jours Accessibilité débutants 45/100
Anbeeld/beellama.cpp#203 · 2 commentaires ·
-
Difficulté 4/5 3-5 jours Accessibilité débutants 45/100
Anbeeld/beellama.cpp#202 · 1 commentaire ·
Toutes les issues de Anbeeld/beellama.cpp
Issues similaires
-
Difficulté 2/5 1-3 heures Accessibilité débutants 72/100
CachyOS/packageinstaller#144 ·
Les mainteneurs répondent en général sous 1 jour
-
documentation
Difficulté 1/5 Moins d'une heure Accessibilité débutants 82/100
TomerAberbach/profiler-md#951 ·
Les mainteneurs répondent en général sous 1 jour
-
[Bug] Default key routing (JavaStringHash) picks a different partition than the Java client for about half of keysPeut-être pris @0z5a l’a pris aujourd’hui. Ouverte
Difficulté 2/5 1-3 heures Accessibilité débutants 82/100
apache/pulsar-client-go#1575 ·
Les mainteneurs répondent en général sous 2 jours
-
Difficulté 2/5 1-3 heures Accessibilité débutants 72/100
pocoproject/poco#5764 ·
Les mainteneurs répondent en général sous 2 jours
-
bug cpp-core-guidelines low-priority
Difficulté 2/5 1-3 heures Accessibilité débutants 62/100
NatronGitHub/openfx-supportext#164 ·
Les mainteneurs répondent en général sous 1 jour