Assertion "(check_gdr_support(dev)) == (true)" failed

Open
  • Is something wrong?
#206 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
20/100
Issue type
Bug
Clarity
Needs clarification
Activity status
Stale
Tech stack
c, debian, linux
Domain
networking, operating-systems

Research direction

Start by reproducing the failures in apiperf and copybw, then inspect the reported gdrdrv nvidia_p2p_get_pages error and the nvshmemHelloWorld.cu failure. Compare the installed MLNX_OFED, nv_peer_mem, GDRcopy, and CUDA setup with the linked requirements and FAQ; done means the GDRcopy tests and nvshmem example run successfully.

Written by the indexing model from the issue text.

Description

question

I'm trying to install nvshmem on a machine with several GeForce RTX 3090, running on Debian 10.10, nvidia-cuda-toolkit from backports (cuda 11.2.2). As indicated by the software requirements of nvshmem [1], I've installed MLNX_OFED (v5.3-1.0.0.1 from source), nv_peer_mem (v1.1-0), and GDRcopy (v2.3), and nvshmem (v2.2.1).

The machine has an HDR (200Gbs) Infiniband card :

# lspci |grep -i mell
c2:00.0 Infiniband controller: Mellanox Technologies MT28908 Family [ConnectX-6]

The installation of all packages went fine. But all tests from GDRcopy fail with this kind of error

$ apiperf 
...
selecting device 0
Assertion "(check_gdr_support(dev)) == (true)" failed at apiperf.cpp:262


# copybw
...
selecting device 0
testing size: 131072
rounded size: 131072
Assertion "(check_gdr_support(dev)) == (true)" failed at copybw.cpp:253

When running these tests, syslog reports this kind of error

Jul 7 13:48:56 myhost kernel: [169336.898388] gdrdrv:__gdrdrv_pin_buffer:nvidia_p2p_get_pages(va=7ff512200000 len=65536 p2p_token=0 va_space=0) failed [ret = -22]

Although using nvshmem fails. Trying to run run the nvshmemHelloWorld.cu example fails with
src/comm/transports/ibrc/ibrc.cpp: NULL value get_device_list failed
I've also tried to install nvshmem without support for GDRcopy, but that does not change anything, GDRcopy seems to be a strict requirement.

I've also followed the FAQ [2], and confirm that nv_peer_mem and gdrdrv are loaded:


#  lsmod | grep nv_peer_mem
nv_peer_mem            16384  0
ib_core               352256  9 rdma_cm,ib_ipoib,nv_peer_mem,iw_cm,ib_umad,rdma_ucm,ib_uverbs,mlx5_ib,ib_cm
nvidia              34119680  198 nvidia_uvm,nv_peer_mem,gdrdrv,nvidia_modeset

#  lsmod | grep gdr
gdrdrv                 24576  0
nvidia              34119680  198 nvidia_uvm,nv_peer_mem,gdrdrv,nvidia_modeset

Do you have any suggestions what could be wrong and how to solve this ?

[1] https://docs.nvidia.com/hpc-sdk/nvshmem/install-guide/index.html
[2] https://docs.nvidia.com/hpc-sdk/nvshmem/api/docs/faq.html

Dominant language
C
Stars
1.4k
Forks
198
No merged PRs in 30d

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from NVIDIA/gdrcopy

All issues in NVIDIA/gdrcopy

Similar issues

More C issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.