i2_s AVX2 GEMM folds the int16 accumulator 10x more often than its own bound requires (7.5% measured)
还没有人认领这个 Issue。
评估
- 难度
- 4/5
- 预计耗时
- 3-5 天
- 新手友好度
- 55/100
- Issue 类型
- 重构
- 描述清晰度
- 基本清楚
- 活跃度
- 活跃
- 技术栈
- cpp
- 领域
- performance
调研方向
从 ggml/src/ggml-cpu/llamafile/sgemm.cpp 中的 non-VNNI tinyBLAS_I2S_AVX::gemm 路径开始，然后与 ggml/src/ggml-cpu/ggml-cpu-i2s.c 中的 ggml_gemm_i2_i8_s 进行比较。阅读 ggml/src/ggml-cpu/arch/x86/quants.c 中对 TQ2_0 的处理，并对 non-VNNI 路径进行前后 benchmark。完成的标准是两处都延后安全的 int16 folding，同时不改变 VNNI 行为或困惑度。
由索引模型根据 Issue 内容生成。
描述
Summary
tinyBLAS_I2S_AVX::gemm folds its int16 accumulator into int32 once per
128-weight block. Its own operands license once per ten. Deferring the fold
is worth a measured 7.5 % on the dispatched path, with perplexity unchanged.
This is the path that actually runs: on a default build the per-row vec_dot is
never called (calls=0 sgemm=52068 from the kernel's own counters), so the
saving is on the hot path rather than a fallback.
The bound, from the kernel's own operands
ggml/src/ggml-cpu/llamafile/sgemm.cpp, non-VNNI branch:
__m256i d0 = _mm256_maddubs_epi16(a_vals[r][0], b0);
__m256i d1 = _mm256_maddubs_epi16(a_vals[r][1], b1);
__m256i d2 = _mm256_maddubs_epi16(a_vals[r][2], b2);
__m256i d3 = _mm256_maddubs_epi16(a_vals[r][3], b3);
__m256i sum = _mm256_add_epi16(_mm256_add_epi16(d0, d1),
_mm256_add_epi16(d2, d3));
acc[r][c] = _mm256_add_epi32(acc[r][c],
_mm256_madd_epi16(sum, one16)); // every block
Codes are masked to 0..3 (the lut comment a few lines above says so) and
activations are int8, so one vpmaddubsw lane takes two products of at most
3 * 127:
per lane, per plane 2 * 3 * 127 = 762
four planes per block 4 * 762 = 3048
safe fold interval floor(32767 / 3048) = 10 blocks
With the codes an encoder actually emits (0..2; code 3 measured absent in
2,084,044,800 weights of BitNet-b1.58-2B-4T) the bound is 16. Ten is the
conservative figure and is what I measured against.
Measurement
Ryzen 5 3600 (Zen 2, AVX2, no VNNI), same binary, same session, mean of three:
folding every block 10,254,922,296 cycles
folding every ten 9,488,166,828 cycles
difference 7.5 %
llama-perplexity over the same corpus is identical between the two builds, so
this is a cost and not a trade.
Scope
- Non-VNNI only. The
__AVXVNNI__/__AVX512VNNI__branch above uses
_mm256_dpbusd_epi32, which accumulates straight toint32; there is no fold
to amortise and nothing to gain there. - A second site has the same shape and I have not benchmarked it:
ggml/src/ggml-cpu/ggml-cpu-i2s.c, inggml_gemm_i2_i8_s, builds the same
sum16from fourmaddubsand folds it per block. A fix should probably
cover both.
Suggested shape of a fix
Hoist the madd_epi16 out of the block loop and run an int16 accumulator for
up to ten blocks, folding at the boundary and at the tail. Upstream ggml does
exactly this for its own ternary types and documents the reasoning in
ggml/src/ggml-cpu/arch/x86/quants.c — TQ2_0 carries the comment
// 16-bit sums, because 256*127 still fits. That is the idiom this kernel is
missing, and the best reference for the change.
Provenance
Found while measuring this kernel as a baseline for unrelated work. The 7.5 %
figure, the bound derivation and the calls=0 sgemm=52068 dispatch counts are
reproducible from https://github.com/purpleskulll/bitnet-t5b — results/i2s_fold_interval.txt
carries the run. I have no patch to offer that I have tested against your CI;
happy to prepare one if the direction is welcome.
I searched issues and PRs for fold, accumulator, int16, tinyBLAS_I2S,
maddubs and madd_epi16 before opening this and found nothing covering it.
#259 is the nearest neighbour and is a different kernel and a different idea.
- 主要语言
- C++
- 星标
- 40.3k
- 派生
- 3.7k
- PR 合并指标
- 30 天内没有已合并 PR
环境准备
这个项目没有提供开发容器、Dockerfile 或贡献指南，环境需要你自己搭建：先看它的 README，通用步骤见我们的新手贡献指南。
从这里开始
- 先读完整个 Issue，再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库，在一个分支上完成修改。
- 提交 Pull Request，并在描述里引用这个 Issue 编号。
microsoft/BitNet 的其他 Issue
-
难度 2/5 1-3 小时 新手友好度 86/100
-
i2_s SIGSEGVs at n_ubatch >= 32: BLAS backend dequantises by a row stride 4x the real packed row未关闭
难度 2/5 1-3 小时 新手友好度 76/100
-
难度 2/5 1-3 小时 新手友好度 86/100
-
难度 1/5 1 小时以内 新手友好度 92/100
-
难度 2/5 1-3 小时 新手友好度 76/100
相似的 Issue
-
bug
难度 2/5 1-3 小时 新手友好度 84/100
维护者通常 1 天内回复
-
ai_p2
难度 2/5 1-3 小时 新手友好度 76/100
ClickHouse/ClickHouse#123351 ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 88/100
stillwater-sc/universal#1617 · 1 条评论 ·
维护者通常 1 天内回复
-
难度 1/5 1 小时以内 新手友好度 78/100
keepassxreboot/keepassxc#13737 ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 72/100
Vector35/binaryninja-api#8621 · 1 条评论 ·
维护者通常 2 天内回复