[BUG] pdlp_class.run_double_very_low_accuracy never converges on GH200 and hangs conda-cpp-tests

Abierto
  • ¿Algo no está bien?
#2,136 2 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
15/100
Tipo de issue
Error
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
cpp
Área
performance, testing-qa

Línea de trabajo

The failing case is pdlp_class.run_double_very_low_accuracy in cpp/tests/linear_programming/pdlp_test.cu, which runs PDLP on afiro with PSLP presolve and 1e-12 absolute tolerances. Reproducing needs GH200 (arm64, Hopper) hardware, which a newcomer probably lacks, so start by reading the test and the PDLP iteration code it exercises. Done means the test has an explicit time limit so it fails fast, and the divergence on GH200 is either fixed or narrowed to a cause.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

bug pdlp

Summary

On the GH200 runner (arm64, CUDA 13.3, ubuntu26.04), pdlp_class.run_double_very_low_accuracy never converges. The PDLP iteration diverges from the very first checkpoint and runs until the 90 minute gtest limit kills the job, so conda-cpp-tests fails on that runner most nights.

Evidence

  • It was the last test started in 7 of the last 8 nightly failures of that job (conda-cpp-tests / 13.3.0, 3.14, arm64, ubuntu26.04, gh200), e.g. run 38520012990 job 115707369040. The 8th (10-09) stopped on pdlp_class.simple_batch_different_constraint_bounds, which I did not look at.
  • The test (cpp/tests/linear_programming/pdlp_test.cu) solves afiro with PDLP only, all absolute tolerances at minimal_absolute_tolerance (1e-12), relative tolerances 0, and no time limit.
  • Same input on an x86 RTX PRO 6000 (CUDA 13.3): PSLP presolves to 14 rows and 18 columns, converges at iteration 1500 in 0.078 s. On the GH200 the PSLP-presolved problem is identical, but at iteration 1000 the primal and dual objective are −482.9 and −468.0 (gap 14.9) instead of −464.753143 for both, and the residuals then oscillate around 1e-6 to 1e-7 for 286 million iterations (about 4939 s at the point the log stops).
  • On the RTX PRO 6000 and L4 jobs in the same run the test passes in 71 to 79 ms. The other GH200-free jobs in that run passed the whole suite.

I could not reproduce it on the hardware I have (amd64 Quadro RTX 8000), so the cause is open. It could be arm64, Hopper-with-Grace, or CUDA 13.3 specific.

Proposed

  1. Give the test a time limit so it fails fast instead of consuming the whole 90 minute budget and hiding every later test.
  2. Investigate the PDLP divergence on GH200: compare the first iterations against x86, check for uninitialized memory or a race, and try the earlier nightlies.

#2052 covers other arm64 conda-cpp-tests hangs; this one is deterministic and tied to this test.

Lenguaje dominante
Cuda
Estrellas
1.1k
Forks
249
Merge medio
1 d 15 h
PR fusionados (30 d)
126

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de NVIDIA/cuopt

Todos los issues de NVIDIA/cuopt

Issues similares

Más issues de Performance

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.