DeepSeek-V4.1's fp4_e4m3_b16 pack rounds a tied scale up on pre-sm_89 GPUs
维护者通常 1 天内回复
还没有人认领这个 Issue。
评估
- 难度
- 4/5
- 预计耗时
- 半天
- 新手友好度
- 42/100
- Issue 类型
- 缺陷
- 描述清晰度
- 描述清楚
- 活跃度
- 活跃
调研方向
Start with test_roundtrip_matches_the_reference_quantizer in tests/kernels/test_dsv41_pack.py, which fails for the fp4_e4m3_b16 cases at 128 and 512. Then read _pack_rows_kernel and round_e4m3 in the kernel module, where the scale is computed as amax * (1/6) and the subnormal rounding is fused into an add. Run the test with FREETOKEN_FORCE_E4M3_EMU=1 on a GPU to reproduce. Done means both cases pass and the scales match torch's cast on tied midpoints.
由索引模型根据 Issue 内容生成。
描述
Before you start
- I have read the FAQ and my problem is not answered there.
- I have read the Roadmap and this is not already planned there.
- I have searched existing issues and found no duplicate.
- I am on the latest release, or on a freshly rebuilt
mainwhen building from source.
What happened
On an RTX 3090 Ti (sm_86), test_roundtrip_matches_the_reference_quantizer[128-fp4_e4m3_b16] and [512-fp4_e4m3_b16] in tests/kernels/test_dsv41_pack.py fail with fp4_e4m3_b16: 0.015625. The kernel stores some group scales one e4m3 step above the reference's: 2 of 2,400 groups at dim 128 and 16 of 9,600 at dim 512. In every one, amax / 6 lies exactly on an e4m3 midpoint. For example, amax 0.052734375 gives 0.0087890625, halfway between the subnormals 0.0078125 and 0.009765625. .to(torch.float8_e4m3fn) rounds to the even 0.0078125, but the kernel stores 0.009765625.
The emulated path does this. _pack_rows_kernel rounds the scale with _e4m3_round(amax * (1.0 / 6.0)), and round_e4m3's subnormal branch is (x + 24576.0) - 24576.0. Triton fuses the multiply into that add: a kernel computing round_e4m3(amax * (1.0 / 6.0)) compiles the add to fma.rn.f32 %r8, %r1, 0f3E2AAAAB, 0f46C00000. The add therefore rounds the unrounded product, which lies just above the midpoint because 1/6 rounds up in fp32, and gets the odd code. The same value loaded from memory rounds to 0.0078125. The native branch converts in hardware and does not take this path.
I tested three fixes here; each rounds both tied values I hit (amax 0.052734375 and 0.076171875) as torch's cast does:
- Divide as the reference does:
s = _e4m3_round(tl.math.div_rn(amax, 6.0)), as the kernel already does for the codes. With this change the test file passes, 10 of 10. - Make
round_e4m3's subnormal branch unfusable withlibdevice.sub_rn(libdevice.add_rn(x, 24576.0), 24576.0). That keeps its round-to-nearest-even result whatever the caller computes its argument with. The other callers clamp before rounding, so no multiply reaches the add there today. - Compile with
enable_fp_fusion=False. Triton 3.9 turns fusion off by default (#629), which would hide this.
How did you install FreeToken
Built from source
FreeToken version
9c635a1
OS
WSL2 on Windows 11
OS details
Ubuntu 24.04, Python 3.12, torch 2.14.0+cu130, triton 3.8.0. Native Windows 11 (Python 3.13, triton-windows 3.8.0, with the Windows PRs from #384) gives the same scales.
GPU and driver
RTX 3090 Ti 24 GB, driver 617.14
CPU and system RAM
AMD Ryzen 9 9950X3D, 256 GB
Checkpoint
None; the kernel test runs without one.
Command
pytest -q --tb=line -rf tests/kernels/test_dsv41_pack.py
Full log
..F...F... [100%]
=================================== FAILURES ===================================
/home/yevhenii/FreeToken/tests/kernels/test_dsv41_pack.py:73: AssertionError: fp4_e4m3_b16: 0.015625
/home/yevhenii/FreeToken/tests/kernels/test_dsv41_pack.py:73: AssertionError: fp4_e4m3_b16: 0.015625
=========================== short test summary info ============================
FAILED tests/kernels/test_dsv41_pack.py::test_roundtrip_matches_the_reference_quantizer[128-fp4_e4m3_b16]
FAILED tests/kernels/test_dsv41_pack.py::test_roundtrip_matches_the_reference_quantizer[512-fp4_e4m3_b16]
2 failed, 8 passed in 3.93s
Anything else
FREETOKEN_FORCE_E4M3_EMU=1 selects the same emulated path, so it should reproduce on a newer GPU too; I could only run it on sm_86.
- 主要语言
- Python
- 星标
- 14.2k
- 派生
- 1.4k
- 平均合并
- 21 小时 39 分钟
- 30 天内合并 PR
- 100
环境准备
这个项目没有提供开发容器、Dockerfile 或贡献指南，环境需要你自己搭建：先看它的 README，通用步骤见我们的新手贡献指南。
从这里开始
- 先读完整个 Issue，再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库，在一个分支上完成修改。
- 提交 Pull Request，并在描述里引用这个 Issue 编号。
FlashML-org/FreeToken 的其他 Issue
-
Desktop feature
难度 2/5 1-3 小时 新手友好度 72/100
FlashML-org/FreeToken#800 · 1 条评论 ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 84/100
FlashML-org/FreeToken#767 ·
维护者通常 1 天内回复
-
test_gemv_path_matches_the_transcription fails on sm_120 (bf16 rounding tie exceeds the tolerance)可能已有人在做 @void-mckenzie 于 17 天前认领。 未关闭
难度 2/5 1-3 小时 新手友好度 77/100
FlashML-org/FreeToken#649 · 1 条评论 ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 78/100
FlashML-org/FreeToken#648 · 1 条评论 ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 76/100
FlashML-org/FreeToken#646 · 1 条评论 ·
维护者通常 1 天内回复
查看 FlashML-org/FreeToken 的全部 Issue
相似的 Issue
-
难度 2/5 1-3 小时 新手友好度 68/100
-
难度 2/5 1-3 小时 新手友好度 70/100
维护者通常 2 天内回复
-
难度 2/5 1-3 小时 新手友好度 62/100
维护者通常 2 天内回复
-
bug
难度 2/5 1-3 小时 新手友好度 82/100
developmentseed/titiler-cmr#278 ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 82/100
维护者通常 2 天内回复