[BUG] conda-cpp-tests on gb300: routing PDP test hits illegal memory access, set_shmem_of_kernel fails after it

Aperta
  • Qualcosa non va?
#2,212 1 commento 0 reazioni 1 assegnatario Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

@0z5a ci sta già lavorando.

Dal 3/11/2026.

  • #2260 di @0z5a — aperta

Valutazione

Difficoltà
4/5
Tempo stimato
1-2 giorni
Idoneità per principianti
48/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Ambito
backend, testing-qa

Direzione di ricerca

Start with the failing cost_boundary_initialization.pdp_set_route_data_resets_cost_boundaries test and the set_shmem_of_kernel tests in the routing test binary; the report says the latter failures may be fallout from a sticky CUDA error. Check the nightly run and reproduce on gb300 if available, then trace the illegal memory access in the PDP test. Done when the PDP test passes and the shared memory tests no longer see error 700.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

bug

Describe the bug
In nightly run 40167716943 (2026-10-25, b5ad4d07), conda-cpp-tests / 13.3.0, 3.14, arm64, ubuntu26.04, gb300, latest-driver, latest-deps fails (job 121092087617). 3 routing tests fail, in every retry:

  • cost_boundary_initialization.pdp_set_route_data_resets_cost_boundaries: C++ exception from cuda/__driver/driver_api.h:662, an illegal memory access was encountered
  • set_shmem_of_kernel.zero_request: cudaGetLastError() returns 700 (cudaErrorIllegalAddress), expected cudaSuccess (set_shmem_of_kernel.cu:32)
  • set_shmem_of_kernel.normal_request: set_shmem_of_kernel(kernel_normal, 4096) returns false (set_shmem_of_kernel.cu:38)

The two set_shmem_of_kernel failures look like fallout of the illegal access in the preceding PDP test: the sticky error 700 is still set when they run. Within the routing test binary, the other 57 tests pass.

  • All 12 other conda-cpp-tests jobs in the run passed.
  • The same commit passed on gb300 the next night (40277562656), so this is intermittent.
  • This is not the failure set in #2176 (pdlp_class, Population, ReducedMatrix). None of those tests failed in this run.

Expected behavior
cost_boundary_initialization.pdp_set_route_data_resets_cost_boundaries passes on gb300 as on the other GPUs.

Environment details

  • Environment location: CI, gb300 arm64, CUDA 13.3.0, Python 3.14, ubuntu26.04, latest driver
  • Method of cuOpt install: conda build from source (rattler-build)

Additional context
Related: #2176 (other gb300-only C++ test failures).

Lingua principale
Cuda
Stelle
1.1k
Fork
249
Merge medio
1g 15h
PR unite (30g)
126

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di NVIDIA/cuopt

Tutte le issue di NVIDIA/cuopt

Issue simili

Altre issue su Backend & API Design

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.