cdi generate: a per-function spec for ConnectX network functions (nvidia.com/nic)

Aperta
  • Qualcosa non va?
#2,197 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
35/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
go, linux
Ambito
infrastructure

Direzione di ricerca

Start from the existing --mode mofed path in nvidia-ctk cdi generate, which the issue names as the closest existing mode, and add a nic mode beside it. Walk the sysfs paths listed in the issue (/sys/bus/pci/devices/<bdf>/infiniband, /sys/class/infiniband_verbs, /sys/class/infiniband_mad) to build the per-function devices, sorted by PCI address. Done when the generated spec for a host with several ConnectX functions matches the JSON example, including Ethernet-only functions with an env entry and no device nodes.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Why

Kata runtime-rs annotates every ConnectX function it passes through to a VM as
cdi.k8s.io/vfio<N>: nvidia.com/nic=<index>, indices in guest PCI order
(src/runtime-rs/crates/resource/src/cdi_devices/mod.rs, row 0x15b3-0x020).
kata-agent resolves those names against /var/run/cdi before it creates the
container and fails the container after 100 s without a match. Nothing in the
guest writes a spec of that kind today, so any pod with a ConnectX PF or VF,
and the HGX Service VM holding the four NVSwitch management PFs, cannot start
its container under runtime-rs.

The GPU side has this covered: the guest init runs nvidia-ctk cdi generate
and the agent resolves nvidia.com/gpu=<index>. The NIC side needs the same
producer. --mode mofed is not it: it emits kind nvidia.com/gpu, one device
named all, every /dev/infiniband/uverbs* plus rdma_cm, no per-function
devices and no indices.

Ask

A discovery mode nic (name open) that emits, for every PCI function with
vendor 0x15b3 and base class 0x02 (Ethernet 0x0200 and InfiniBand
0x0207 alike, PF or VF), sorted by PCI address:

{
  "cdiVersion": "0.6.0",
  "kind": "nvidia.com/nic",
  "devices": [
    {"name": "0", "containerEdits": {"env": ["NVIDIA_NIC_0=0000:04:00.0"],
      "deviceNodes": [{"path": "/dev/infiniband/issm0"}, {"path": "/dev/infiniband/rdma_cm"},
                      {"path": "/dev/infiniband/umad0"}, {"path": "/dev/infiniband/uverbs0"}]}},
    {"name": "1", "containerEdits": {"env": ["NVIDIA_NIC_1=0000:05:00.0"], "deviceNodes": ["..."]}}
  ]
}
  • Device <index> is the function's position in PCI-address order; that is the
    order runtime-rs uses (sort_devices_by_guest_pcipath), and the same per-VM
    index contract the GPU spec already relies on.
  • Nodes come from the function's InfiniBand devices:
    /sys/bus/pci/devices/<bdf>/infiniband/<ibdev>, then every
    /sys/class/infiniband_verbs/uverbs*/ibdev and /sys/class/infiniband_mad/{umad,issm}*/ibdev
    equal to <ibdev>, plus /dev/infiniband/rdma_cm when present. A function
    without an InfiniBand device (Ethernet with RDMA off) still gets its device
    entry, so the index set stays complete; the env entry keeps the device valid
    (CDI rejects a device with no edits) and tells the container which function
    device <index> is.
  • No NVML requirement: the Service VM has no GPU. --mode mofed already runs
    without a driver ("Could not determine driver version ... ignoring").

Verified shape

A guest-side generator producing exactly this document was run on an HGX B300
with kata runtime-rs's row in place: the agent resolved four devices, injected
all of them, the container was Running 49 s after apply, and PID 1 carried
NVIDIA_NIC_0..3 with the guest addresses in root-port order. The document
above is that output. The generator is a bridge in the guest init (NVIDIA/nvrc#245)
and should be replaced by this toolkit mode.

Lingua principale
Go
Stelle
4.6k
Fork
612
Merge medio
2g 14h
PR unite (30g)
41

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di NVIDIA/nvidia-container-toolkit

Tutte le issue di NVIDIA/nvidia-container-toolkit

Issue simili

Altre issue su Go

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.