[BUG] pdlp_class.run_double_very_low_accuracy never converges on GH200 and hangs conda-cpp-tests
维护者通常 1 天内回复
还没有人认领这个 Issue。
评估
- 难度
- 4/5
- 预计耗时
- 3-5 天
- 新手友好度
- 15/100
- Issue 类型
- 缺陷
- 描述清晰度
- 基本清楚
- 活跃度
- 活跃
- 技术栈
- cpp
调研方向
The failing case is pdlp_class.run_double_very_low_accuracy in cpp/tests/linear_programming/pdlp_test.cu, which runs PDLP on afiro with PSLP presolve and 1e-12 absolute tolerances. Reproducing needs GH200 (arm64, Hopper) hardware, which a newcomer probably lacks, so start by reading the test and the PDLP iteration code it exercises. Done means the test has an explicit time limit so it fails fast, and the divergence on GH200 is either fixed or narrowed to a cause.
由索引模型根据 Issue 内容生成。
描述
Summary
On the GH200 runner (arm64, CUDA 13.3, ubuntu26.04), pdlp_class.run_double_very_low_accuracy never converges. The PDLP iteration diverges from the very first checkpoint and runs until the 90 minute gtest limit kills the job, so conda-cpp-tests fails on that runner most nights.
Evidence
- It was the last test started in 7 of the last 8 nightly failures of that job (
conda-cpp-tests / 13.3.0, 3.14, arm64, ubuntu26.04, gh200), e.g. run 38520012990 job 115707369040. The 8th (10-09) stopped onpdlp_class.simple_batch_different_constraint_bounds, which I did not look at. - The test (
cpp/tests/linear_programming/pdlp_test.cu) solves afiro with PDLP only, all absolute tolerances atminimal_absolute_tolerance(1e-12), relative tolerances 0, and no time limit. - Same input on an x86 RTX PRO 6000 (CUDA 13.3): PSLP presolves to 14 rows and 18 columns, converges at iteration 1500 in 0.078 s. On the GH200 the PSLP-presolved problem is identical, but at iteration 1000 the primal and dual objective are −482.9 and −468.0 (gap 14.9) instead of −464.753143 for both, and the residuals then oscillate around 1e-6 to 1e-7 for 286 million iterations (about 4939 s at the point the log stops).
- On the RTX PRO 6000 and L4 jobs in the same run the test passes in 71 to 79 ms. The other GH200-free jobs in that run passed the whole suite.
I could not reproduce it on the hardware I have (amd64 Quadro RTX 8000), so the cause is open. It could be arm64, Hopper-with-Grace, or CUDA 13.3 specific.
Proposed
- Give the test a time limit so it fails fast instead of consuming the whole 90 minute budget and hiding every later test.
- Investigate the PDLP divergence on GH200: compare the first iterations against x86, check for uninitialized memory or a race, and try the earlier nightlies.
#2052 covers other arm64 conda-cpp-tests hangs; this one is deterministic and tied to this test.
- 主要语言
- Cuda
- 星标
- 1.1k
- 派生
- 249
- 平均合并
- 1 天 15 小时
- 30 天内合并 PR
- 126
环境准备
- 没有 Dockerfile 或 Docker Compose 文件
- 有 Pull Request 模板
- 阅读贡献指南
从这里开始
- 先读完整个 Issue，再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库，在一个分支上完成修改。
- 提交 Pull Request，并在描述里引用这个 Issue 编号。
NVIDIA/cuopt 的其他 Issue
-
难度 2/5 1-3 小时 新手友好度 72/100
维护者通常 1 天内回复
-
难度 1/5 1 小时以内 新手友好度 82/100
维护者通常 1 天内回复
-
难度 4/5 3-5 天 新手友好度 32/100
维护者通常 1 天内回复
-
CI: nightly Cython build with bounds checking enabled可能已有人在做 关联的 PR 仍在进行中或已合并。 未关闭
难度 4/5 3-5 天 新手友好度 22/100
NVIDIA/cuopt#2255 · 1 个 reaction ·
维护者通常 1 天内回复
-
难度 3/5 半天 新手友好度 50/100
维护者通常 1 天内回复
相似的 Issue
-
area:mlx difficulty:2 enhancement
难度 2/5 1-3 小时 新手友好度 75/100
MakazhanAlpamys/Soup#2490 ·
维护者通常 1 天内回复
-
[BUG] Segfault when all runs of a Complexity() benchmark are skipped可能已有人在做 @nitanshnagpal 今天认领。 未关闭
难度 2/5 1-3 小时 新手友好度 75/100
维护者通常 1 天内回复
-
Code: Performance
难度 1/5 1 小时以内 新手友好度 88/100
CrimsonCrossBunker/Cataclysm-Cleanwater-Bomb#1167 ·
维护者通常 1 天内回复
-
POSIX-Only Path Normalization in `base()` Breaks Project Ledgers and Bypasses Transcript Cache on Windows可能已有人在做 关联的 PR 仍在进行中或已合并。 未关闭
难度 2/5 1-3 小时 新手友好度 78/100
DietrichGebert/ponytail#1342 ·
维护者通常 1 天内回复
-
[BUG]: instrumentation.ServiceNameWithSource rebuilds its tracer.Tag option each time it is applied未关闭needs-triage
难度 2/5 1-3 小时 新手友好度 75/100
DataDog/dd-trace-go#5811 ·
维护者通常 2 天内回复