Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

[BUG] CAGRA hang on h100 oldest-deps CI (cuvs test_cagra, then cuvs_bench test_run_command_creates_results)

Aperta
  • Qualcosa non va?
#2,772 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 2 giorni

@bdice ci sta già lavorando.

Dal 11/10/2026.

  • #2829 di @bdice — aperta

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
42/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
python
Ambito
machine-learning, testing-qa

Direzione di ricerca

Start with tests/test_cagra.py, especially the suspected test_cagra_dataset_dtype_host_device area, and tests/test_cli.py::test_run_command_creates_results; inspect the failing CI logs and the ci/test_python.sh timeout setup. Reproduce on the specified H100, CUDA 12.2, oldest-deps matrix and capture where each process hangs. Done means the intermittent hangs are explained and addressed, with both test suites passing reliably; the separate symbol-lookup failure is out of scope.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

tests/test_cli.py::test_run_command_creates_results in cuvs_bench failed in 4 pr.yaml runs since 2026-10-06, all on the 12.2.2, 3.11, amd64, rockylinux8, h100, earliest-driver, oldest-deps matrix entry. test_plot_command_creates_png_files fails in the same jobs because it reads the results the run test did not produce.

The Aborted! message is not a crash. It is click's response to SIGINT: ci/test_python.sh runs pytest under timeout 40m --signal=SIGINT, and the benchmark was hung until timeout fired.

In all 4 jobs:

  1. The pytest cuvs step hits its 40 minute timeout and is SIGKILLed. It stalls inside tests/test_cagra.py at the same point (after roughly 52 tests, likely in test_cagra_dataset_dtype_host_device).
  2. The pytest cuvs-bench step then starts, and CUVS_CAGRA_HNSWLIB_ANN_BENCH hangs about 3 seconds in, right after "We have sufficient memory to proceed with in memory build", until its own 40 minute timeout.

A passing run of the same job finishes the cuvs suite in 11 minutes and the bench suite in 72 seconds, with the same driver (580.178.04), CUDA 12.2 and rmm version, so the hang is intermittent. Two of the failing commits already contain #2729. The root cause is not yet identified; the logs do not show where the process is stuck.

Failing jobs:

PR Job
#2649 https://github.com/NVIDIA/cuvs/actions/runs/37562048604/job/113140366555
#2649 https://github.com/NVIDIA/cuvs/actions/runs/37526496924/job/112498097520
#2753 https://github.com/NVIDIA/cuvs/actions/runs/37513765252/job/112456033858
#2429 https://github.com/NVIDIA/cuvs/actions/runs/37426744344/job/112159622042

A fifth job, https://github.com/NVIDIA/cuvs/actions/runs/34719781952/job/104185737619 (PR #2600, CUDA 13.3.0, rtxpro6000), failed the same two tests in 5.8 seconds with symbol lookup error on the bench binaries. That is a separate build/ABI problem, not this hang.

Method: failed pr.yaml runs created on or after 2026-09-08 (230 of 555), then failed python-tests jobs, then FAILED lines in their logs. The count may be incomplete if some job listings could not be fetched.

Lingua principale
Cuda
Stelle
869
Fork
243
Merge medio
3g 11h
PR unite (30g)
73

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di NVIDIA/cuvs

Tutte le issue di NVIDIA/cuvs

Issue simili

Altre issue su Machine Learning

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.