test_backend_cross_device: MoeRouterTopK flakes intermittently on gfx1200
维护者通常 1 天内回复
还没有人认领这个 Issue。
评估
- 难度
- 4/5
- 预计耗时
- 3-5 天
- 新手友好度
- 45/100
- Issue 类型
- 缺陷
- 描述清晰度
- 基本清楚
- 活跃度
- 活跃
- 技术栈
- cpp
- 领域
- testing-qa
调研方向
从 tests/vt/test_backend_cross_device.cpp:2304 的 MoeRouterTopK 测试开始，并比较在 gfx1200 上单独运行与重复完整运行 test_backend_cross_device 二进制文件的结果。调查间歇性出现的 NMSE 和 expert-index 不匹配，包括 allocator 或 queue 状态是否使此案例与其他已报告的 ROCm flakes 相关。确定原因并记录修复方案或可执行的结论后，即视为完成。
由索引模型根据 Issue 内容生成。
描述
Row: BACKEND-ROCM
tests/vt/test_backend_cross_device.cpp:2304, TEST_CASE("MoeRouterTopK matches the CPU oracle (f32 and bf16 logits)"), failed once during work on a fresh implementer's row (row/KERNEL-QUANT-CIQ-GEMM-ROCM-IQUANT-w1, unrelated to MoE routing), on isravale (RX 9060 XT, gfx1200, ROCm 7.2.3):
tests/vt/test_backend_cross_device.cpp:2351: ERROR: CHECK( Nmse(ref_w, dw.Download()) <= kNmseTol ) is NOT correct!
values: CHECK( 0.214825 <= 0.0005 )
plus mismatched expert indices in the same case.
Reproduction rate is low and I want to record it honestly rather than overstate it. The implementer who found it saw 1 failure in 12 full-test_backend_cross_device-binary runs with their (unrelated) change applied, and 1 in 12 with that change stashed at the pristine base commit c6e1092dffc5f3484d9c8d24ad3b4e3f7b7531cc — ruling their own diff out as the cause. I (operator) then ran the isolated case 10/10 clean and the full binary 6/6 clean, so 0 reproductions in 16 of my own runs. Net: 2 failures observed in 28 total runs across two sessions (~7%), never in isolation from either session — it may require the full binary's accumulated device/allocator state to trigger, which is why isolated single-case runs may under-sample it.
Confirmed unrelated to any in-flight PR: the test case and vt::MoeRouterTopK's kernel source are both absent from row/KERNEL-QUANT-CIQ-GEMM-ROCM-IQUANT-w1's diff, and the test case is confirmed present, unmodified, at that branch's pre-reconciliation head 22800690b (predates all work on that row).
Distinct from #3119 (causal conv1d, a different test case, much higher ~37% isolated reproduction rate) and from #1954 (MoeSiluMul). Given this project's position that no device-side ASan exists for RDNA/gfx12, and that this session alone has found two independent classes of adjacent-but-different intermittent ROCm numerical flakes, this may be worth checking whether all three share a root cause (a common allocator/queue-state issue) rather than being investigated as three unrelated kernels — but that is a hypothesis, not a finding, and needs someone to actually look.
FOLLOWING_AGENTS_PROTOCOL
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [Claude Code]
- 主要语言
- C++
- 星标
- 423
- 派生
- 53
- 平均合并
- 1 天 7 小时
- 30 天内合并 PR
- 382
环境准备
从这里开始
- 先读完整个 Issue，再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库，在一个分支上完成修改。
- 提交 Pull Request，并在描述里引用这个 Issue 编号。
mudler/vllm.cpp 的其他 Issue
-
难度 2/5 1-3 小时 新手友好度 84/100
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 82/100
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 70/100
维护者通常 1 天内回复
-
难度 1/5 1 小时以内 新手友好度 88/100
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 84/100
维护者通常 1 天内回复
相似的 Issue
-
难度 1/5 1 小时以内 新手友好度 88/100
tenstorrent/tt-metal#58057 · 1 条评论 ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 84/100
maplibre/maplibre-native#4690 ·
维护者通常 1 天内回复
-
comp-query-execution
难度 2/5 1-3 小时 新手友好度 84/100
ClickHouse/ClickHouse#122569 ·
维护者通常 1 天内回复
-
难度 1/5 1 小时以内 新手友好度 92/100
维护者通常 1 天内回复
-
难度 1/5 1 小时以内 新手友好度 88/100