Research: SINQ / A-SINQ on Qwen3.8-27B with the Q3_K_XL recipe — results

Ouverte
#199 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub

Personne n'a encore pris cette issue.

Évaluation

Difficulté
4/5
Temps estimé
3-5 jours
Accessibilité débutants
35/100
Type d'issue
Documentation
Clarté
Plutôt claire
Activité
Active
Stack technique
cpp, python

Piste de recherche

Start by reviewing the SINQ/A-SINQ measurements and practical notes in the issue, then inspect tools/sinq/export_activations.py for the reported MTP-layer and CPU Triton failures. The report documents local workarounds but does not include exact commands or calibration data; clarify the reproducible steps and validate any exporter findings before documenting them.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Description

Research Stage
  • Background Research (Let's try to avoid reinventing the wheel)
  • Hypothesis Formed (How do you think this will work and it's effect?)
  • Strategy / Implementation Forming
  • Analysis of results
  • Debrief / Documentation (So people in the future can learn from us)
Analysis

I tried the SINQ / A-SINQ implementation from the v0.5.0 tag (f0beb9f60, i.e. before the offline optimizer refactor, so the old --sinq-calibration path) on Qwen3.8-27B, reproducing unsloth's UD-Q3_K_XL recipe exactly (504 per-tensor types from the original GGUF, output Q5_K, token_embd Q3_K, unsloth's published imatrix). The plain control is byte-identical to upstream llama.cpp llama-quantize with the same arguments, so the baseline is clean.

Variants (all 13.15 GB, control 13.15 GB):

  • Control: imatrix only
  • SINQ: --sinq-precondition=sidecar, 468 tensors
  • A-SINQ 8k: calibration = 8 × 1024 tokens from my own coding-agent sessions (chat template, tool calls), 128 rows
  • A-SINQ BF16: calibration = 12 × 2048 tokens where the BF16 model wrote the answers itself (mixed prompts: chat, tools, agentic, reasoning, code review, German, multilingual), 256 rows

Mean KLD vs BF16 (with BOS forced at the beginning of every chunk¹, -ub 128):

Control SINQ A-SINQ 8k A-SINQ BF16 UD-Q3_K_XL
wikitext-2, 20 × 4096 0.0399 0.0402 0.0421 0.0377 0.0305
chat conversation, 7 × 2048 0.0615 0.0590 0.0611 0.0532 0.0513
django source, 20 × 4096 0.0600 0.0601 0.0619 0.0605 0.0630
code (coderppl), 5 × 16384 0.01026 0.01054 0.01051 0.01022 0.00976
agentic traces, 8 × 16384 0.0208 0.0209 0.0215 0.0209 0.0209
bartowski calibration-v6, 20 × 4096 0.168 0.184 0.185 0.130 0.120
agentic same top token 93.69 % 92.48 % 92.34 % 92.44 % 93.59 %

Observations:

  • SINQ on top of an imatrix is neutral to slightly worse here (within ±3 % on five texts, +9 % on calibration-v6).
  • A-SINQ depends entirely on the calibration data. With coding-session calibration it was equal or slightly worse everywhere. With self-generated BF16 text it improves prose and chat clearly vs the control (wikitext −6 %, chat −14 %, calibration-v6 −23 %) while code and agentic stay unchanged.
  • It does not close the gap to unsloth's own UD-Q3_K_XL, which is still better on wikitext (−19 %), calibration-v6 (−8 %), code (−4 %) and chat (−4 %).
  • All SINQ variants lose ~1.2 points of top-token agreement on the agentic traces at equal KLD. Control and UD-Q3_K_XL do not.
  • The local L1 gain does not predict the model-level result. For example, ssm_beta always picks a strong alpha with a ~56 % local L1 gain, but that is measured before the sigmoid, so it may not mean much.

¹ A warning about measuring quality with llama-perplexity on Qwen3.8-27B: the model has no BOS (add_bos_token false), so every perplexity window starts mid-document without any start token. In some windows even BF16 loses track, resulting in abnormal results. Forcing a start token (--override-kv tokenizer.ggml.add_bos_token=bool:true, which uses <|endoftext|>) fixes it, and without it the small method differences flip sign:

  • wikitext: without BOS, A-SINQ BF16 looks worst (+10 % vs control); with BOS it is the best of the four (−6 %).
  • calibration-v6: without BOS, A-SINQ BF16 is equal to the control; with BOS it is −23 %.
  • coderppl: without BOS, the mean KLD is 7–9× higher and the 99.9 % KLD ~25× higher, and A-SINQ 8k looks 25 % worse than the control; with BOS, all variants are within 3 %.
  • Unaffected: UD-Q3_K_XL stays ahead either way, and django and the agentic traces barely change.

This is model-specific: on Qwen3.8-Flash-Next (MoE) the same override slightly hurts (PPL +1 % code, +6 % calibration-v6), and a Llama-based model (Bielik 7B) doesn't care either way. So it's worth checking both variants per model before comparing quant methods by PPL/KLD.

Practical notes:

  • tools/sinq/export_activations.py with a GGUF that contains an MTP layer: block_count, layer types and the tensor check fail, because the GGUF has 65 layers vs 64 in HF. I worked around it in a local copy.
  • An installed flash-linear-attention makes transformers pick Triton kernels on CPU, which fails with "Pointer argument cannot be accessed from Triton". Blocking fla and causal_conv1d in the exporter fixes it.
  • Both exporter points are from f0beb9f60; from the source they look unchanged on the current v0.5.0 branch, but I haven't re-run it there.
  • All quants above were made with the unmodified SINQ/A-SINQ code at f0beb9f60 (my build carries only unrelated local patches).

If you need more details (exact commands, per-chunk numbers, the calibration data) or have any questions, I'm happy to help.

AI Disclosure: The analysis, measurements and this text were prepared with AI assistance (Claude Opus 5.5), reviewed and edited by me.

Langage dominant
C++
Étoiles
1.2k
Forks
80
Merge moyen
4 j 20 h
PR mergées (30 j)
2

Préparer son environnement

Ce projet ne fournit ni conteneur de développement, ni Dockerfile, ni guide de contribution : l'installation est à votre charge. Commencez par son README, et consultez notre guide de la première contribution pour les étapes générales.

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Autres issues de Anbeeld/beellama.cpp

Toutes les issues de Anbeeld/beellama.cpp

Issues similaires

Plus d'issues C++

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.