Investigate large hidden-state differences when independent requests are co-packed (Qwen3.6, no-grad)
维护者通常 1 天内回复
还没有人认领这个 Issue。
评估
- 难度
- 5/5
- 预计耗时
- 一周以上
- 新手友好度
- 20/100
- Issue 类型
- 缺陷
- 描述清晰度
- 需要澄清
- 活跃度
- 活跃
- 技术栈
- python
调研方向
从 /var/tmp/art870-packing-boundary-20260915-physical-tiipnfq4/ 中的 source/decision matrix 以及记录的运行根目录下 offline-result/result.json 中的 strict result 开始。在保留共同的三个 requests 的同时，复现 row-count 与 segment/boundary-treatment 的受控比较；完成的标准是确定差异的来源，或记录数值化评定仍然受阻。
由索引模型根据 Issue 内容生成。
描述
A bounded same-actor no-gradient diagnostic found large hidden-state differences when the same three independent requests were included in a larger pack. This currently blocks claiming numerical qualification for the packing/recovery work in #848 / #870 / #900. It does not establish a regression introduced by #900 or identify an underlying kernel.
Observed September 15, 2026 on one H200, Qwen/Qwen3.6-35B-A3B, candidate trainer runtime SHA256 5df3a1138923faa6705862037cd067d4b8b27cbdb9ef99c1094116efeb237d23, source revision 586fc4a6dbbabc876f6c7a571a86c47d6b064e4a. Two trainable rank-1 LoRA slots and a frozen reference were resident, with restored optimizer state; no backward, optimizer update, inference request or cache-release operation occurred.
The same actor ran A1, A2, B, A3:
- A: original requests 0–2, 32,496 packed/logical rows, one admitted child.
- B: original requests 0–11, 134,591 rows, one admitted child; compare the unchanged first three requests with A.
- A2/A1 and A3/A1 were bit-for-bit identical for all three outputs.
- B/A1 exceeded the unchanged BF16 tolerances (
atol=1e-5,rtol=0.016) for all three outputs, with maximum absolute differences 10.9453125, 8.0 and 9.171875. A3/B also failed. - All twelve B outputs were finite. All nine measured resident adapter/optimizer state digests were equal. Full base-model state was not rehashed by this diagnostic.
The four native stages completed. Original strict offline projection confirms native_completed=true, scalar_complete=true, all_numerical_conditions_passed=false, and zero cache-release calls. Process exit 0 is completion, not numerical acceptance. Exact four owned resource UIDs and two host process groups were independently reconciled absent.
Source investigation: the selected GDN path creates fresh root recurrent/conv state and both packings use variable-length processing. Common requests 1/2 retain their GDN offsets but still differ; request 0 moves to a later offset. Full attention, GDN, MoE, compilation, unmeasured base state and observer effects remain possible locations. No cause has been isolated. A marker-based CPU check of the original packing/projection/spool path passed; that is not proof of native numerical correctness.
Next owned work (Schulman/Peirce): a controlled row-count versus segment/boundary-treatment comparison, preserving the common three requests and original token content. Separate baseline/candidate/main-replica backward diagnostics remain in progress. No tolerance relaxation or art.megatron edit is part of this issue intake.
Durable private evidence (payloads are not attached):
- Run root:
/var/tmp/art870-aaba-native-v1-20260915-root - Strict result:
offline-result/result.json, SHA25697d922966396785661cbac6cf247e5e4081a59ef38216b069599eabc66a4e770 - Root closure:
gpu-root-closure.json, SHA2564421c8d875889813a2cf1248e924cb3e8cb178f4d15248f6e5872de5398154b3 - Full raw terminal log SHA256
a70d1185edaf6277ea10b221d1eb87bf23cd35f57bd22ddcf695e26ed7bc292f; 172 replayed actor occurrences join 86 unique actual records, each appearing twice. - Source/decision matrix:
/var/tmp/art870-packing-boundary-20260915-physical-tiipnfq4/, manifest742797acbdd718a1d2d3ecd447ba3192f5cab66bb41eb578b9247316d83c88ab. - Independent local readback:
/var/tmp/art870-aaba-local-readback-20260915-physical-3qcco7wi/, manifestf6227149e2c58c40c4ffd358eb93e9c33bec2333f0a462fec380e5bef148ceb5.
- 主要语言
- Python
- 星标
- 10.8k
- 派生
- 1k
- 平均合并
- 16 小时 14 分钟
- 30 天内合并 PR
- 148
环境准备
- 没有 Dockerfile 或 Docker Compose 文件
- 没有 Pull Request 模板
- 阅读贡献指南
从这里开始
- 先读完整个 Issue，再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库，在一个分支上完成修改。
- 提交 Pull Request，并在描述里引用这个 Issue 编号。
OpenPipe/ART 的其他 Issue
-
Distributed group_mean casts group ids to float32, losing precision vs non-distributed path可能已有人在做 @OnePunchMonk 于 15 天前认领。 未关闭
难度 2/5 1-3 小时 新手友好度 82/100
维护者通常 1 天内回复
-
Add from_entity parameter to _experimental_fork_checkpoint可能重新可做 关联的 PR 已关闭且未合并。 未关闭
难度 2/5 1-3 小时 新手友好度 72/100
维护者通常 1 天内回复
-
难度 4/5 3-5 天 新手友好度 15/100
维护者通常 1 天内回复
-
Tracking: numerical issues found in an audit of ART main after #1105可能已有人在做 关联的 PR 仍在进行中或已合并。 未关闭
难度 4/5 3-5 天 新手友好度 15/100
维护者通常 1 天内回复
-
Nonfused column LoRA misses input-gradient SUM when TP>1 and sequence parallelism is disabled可能已有人在做 @bradhilton 于 16 天前认领。 未关闭
难度 5/5 一周以上 新手友好度 25/100
维护者通常 1 天内回复
相似的 Issue
-
debt
难度 2/5 1-3 小时 新手友好度 68/100
microsoft/vscode-flake8#601 ·
维护者通常 1 天内回复
-
nonews refactor
难度 2/5 1-3 小时 新手友好度 75/100
Uninett/nav#4255 · 1 条评论 · 1 个 reaction ·
维护者通常 3 天内回复
-
namespace operations
难度 1/5 1 小时以内 新手友好度 70/100
EclipseFdn/open-vsx.org#14421 ·
维护者通常 1 天内回复
-
untriaged
难度 2/5 1-3 小时 新手友好度 78/100
ubuntu/ubuntu-project-docs#850 ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 83/100
维护者通常 1 天内回复