cdi generate: a per-function spec for ConnectX network functions (nvidia.com/nic)

Open
  • Is something wrong?
#2,197 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
35/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
go, linux
Domain
infrastructure

Research direction

Start from the existing --mode mofed path in nvidia-ctk cdi generate, which the issue names as the closest existing mode, and add a nic mode beside it. Walk the sysfs paths listed in the issue (/sys/bus/pci/devices/<bdf>/infiniband, /sys/class/infiniband_verbs, /sys/class/infiniband_mad) to build the per-function devices, sorted by PCI address. Done when the generated spec for a host with several ConnectX functions matches the JSON example, including Ethernet-only functions with an env entry and no device nodes.

Written by the indexing model from the issue text.

Description

Why

Kata runtime-rs annotates every ConnectX function it passes through to a VM as
cdi.k8s.io/vfio<N>: nvidia.com/nic=<index>, indices in guest PCI order
(src/runtime-rs/crates/resource/src/cdi_devices/mod.rs, row 0x15b3-0x020).
kata-agent resolves those names against /var/run/cdi before it creates the
container and fails the container after 100 s without a match. Nothing in the
guest writes a spec of that kind today, so any pod with a ConnectX PF or VF,
and the HGX Service VM holding the four NVSwitch management PFs, cannot start
its container under runtime-rs.

The GPU side has this covered: the guest init runs nvidia-ctk cdi generate
and the agent resolves nvidia.com/gpu=<index>. The NIC side needs the same
producer. --mode mofed is not it: it emits kind nvidia.com/gpu, one device
named all, every /dev/infiniband/uverbs* plus rdma_cm, no per-function
devices and no indices.

Ask

A discovery mode nic (name open) that emits, for every PCI function with
vendor 0x15b3 and base class 0x02 (Ethernet 0x0200 and InfiniBand
0x0207 alike, PF or VF), sorted by PCI address:

{
  "cdiVersion": "0.6.0",
  "kind": "nvidia.com/nic",
  "devices": [
    {"name": "0", "containerEdits": {"env": ["NVIDIA_NIC_0=0000:04:00.0"],
      "deviceNodes": [{"path": "/dev/infiniband/issm0"}, {"path": "/dev/infiniband/rdma_cm"},
                      {"path": "/dev/infiniband/umad0"}, {"path": "/dev/infiniband/uverbs0"}]}},
    {"name": "1", "containerEdits": {"env": ["NVIDIA_NIC_1=0000:05:00.0"], "deviceNodes": ["..."]}}
  ]
}
  • Device <index> is the function's position in PCI-address order; that is the
    order runtime-rs uses (sort_devices_by_guest_pcipath), and the same per-VM
    index contract the GPU spec already relies on.
  • Nodes come from the function's InfiniBand devices:
    /sys/bus/pci/devices/<bdf>/infiniband/<ibdev>, then every
    /sys/class/infiniband_verbs/uverbs*/ibdev and /sys/class/infiniband_mad/{umad,issm}*/ibdev
    equal to <ibdev>, plus /dev/infiniband/rdma_cm when present. A function
    without an InfiniBand device (Ethernet with RDMA off) still gets its device
    entry, so the index set stays complete; the env entry keeps the device valid
    (CDI rejects a device with no edits) and tells the container which function
    device <index> is.
  • No NVML requirement: the Service VM has no GPU. --mode mofed already runs
    without a driver ("Could not determine driver version ... ignoring").

Verified shape

A guest-side generator producing exactly this document was run on an HGX B300
with kata runtime-rs's row in place: the agent resolved four devices, injected
all of them, the container was Running 49 s after apply, and PID 1 carried
NVIDIA_NIC_0..3 with the guest addresses in root-port order. The document
above is that output. The generator is a bridge in the guest init (NVIDIA/nvrc#245)
and should be replaced by this toolkit mode.

Dominant language
Go
Stars
4.6k
Forks
609
Avg merge
3d 8h
Merged PRs (30d)
36

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from NVIDIA/nvidia-container-toolkit

All issues in NVIDIA/nvidia-container-toolkit

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.