Native consumers of the libcuopt-mathopt wheel segfault: solver_settings_t is split across mathopt and client

Aperta
  • Qualcosa non va?
#2,009 4 commenti 0 reazioni 1 assegnatario Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

@ramakrishnap-nv ci sta già lavorando.

Dal 30/9/2026.

Valutazione

Questa issue non è ancora stata valutata.

Descrizione

Description

cuopt_cli as shipped in the libcuopt-mathopt wheel segfaults immediately on startup. The same binary built from the same commit into the libcuopt wheel works.

$ pip install libcuopt_mathopt_cu13-26.10.0-py3-none-manylinux_2_28_x86_64.whl
$ .../site-packages/libcuopt_mathopt/bin/cuopt_cli --help
Segmentation fault (core dumped)     # exit 139, zero bytes of output

$ .../site-packages/libcuopt/bin/cuopt_cli --help
Usage: cuopt_cli [--help] [--version] ...   # exit 0, 96 lines

Found while installing each component wheel on its own in a bare container, against the wheels from the latest green build.yaml run on main (run 36580941794, x86_64, cu13).

It is the binary, not the environment

With both wheels installed in the same container, and the mathopt binary forced onto the metapackage's libraries:

libcuopt/bin/cuopt_cli                   exit=0    lines=96
libcuopt_mathopt/bin/cuopt_cli           exit=139  lines=0
mathopt CLI + LD_LIBRARY_PATH=libcuopt/lib64   exit=139

The two are genuinely different compilations, not the same binary with different metadata:

build-id .text .rodata
libcuopt/bin/cuopt_cli b9c1fdb1... 3100a14cc7034fe1 cf1b26601c8b39d2
libcuopt_mathopt/bin/cuopt_cli 924272b9... 724578ddda193f99 b2b4a865cc6f3d2e

libcuopt_client.so has the same build-id in both wheels; libcuopt_mathopt.so and cuopt_cli differ.

Ruled out

  • Unresolved libraries. ldd reports nothing missing. Every entry resolves, including libcuopt_client.so from the sibling wheel via $ORIGIN/../../libcuopt_client/lib64.
  • Lazy binding. LD_BIND_NOW=1 still segfaults, so every symbol resolves; the crash is in execution, not loading.
  • A missing component. Installing libcuopt-routing alongside makes no difference.
  • Arguments. --help and no arguments both crash, before any output.
  • ABI mismatch between CLI and engine. A matched pair from the same build job crashes just as a mismatched pair does.
  • RUNPATH corruption from patchelf. Both binaries carry the same RUNPATH entries and near-identical LOAD segments.
  • Duplicate OpenMP runtimes. Both wheels vendor libgomp-34324e08.so.1.0.0 under one soname, so only one instance loads.

What is known about the crash

Thread 1 "cuopt_cli" received signal SIGSEGV
#0  __memcpy_avx_unaligned ()
#1  0x000062a0875c245d in ?? ()
#2  0x000062a0875c0c3c in ?? ()
#3  __libc_start_call_main (main=0x62a0875c0320, argc=2, ...)

Two frames inside cuopt_cli between main and a memcpy, with no output produced, so it is early in main. The shipped binary is stripped, so the frames have no names; a debug build is needed to go further.

Why nothing caught it

The component wheels are built and published but never installed or imported in CI. Noted previously in https://github.com/NVIDIA/cuopt/issues/1635#issuecomment-5892294816. A job that installs each wheel in a clean environment and runs cuopt_cli --help would catch this directly.

Impact

cuopt_cli is the reason libcuopt-mathopt declares a console entry point. As shipped, a standalone pip install libcuopt-mathopt provides a CLI that cannot start.

Lingua principale
Cuda
Stelle
1.1k
Fork
249
Merge medio
1g 15h
PR unite (30g)
126

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di NVIDIA/cuopt

Tutte le issue di NVIDIA/cuopt

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.