cdi generate: a per-function spec for ConnectX network functions (nvidia.com/nic)

Abierto
  • ¿Algo no está bien?
#2,197 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
35/100
Tipo de issue
Nueva funcionalidad
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
go, linux
Área
infrastructure

Línea de trabajo

Start from the existing --mode mofed path in nvidia-ctk cdi generate, which the issue names as the closest existing mode, and add a nic mode beside it. Walk the sysfs paths listed in the issue (/sys/bus/pci/devices/<bdf>/infiniband, /sys/class/infiniband_verbs, /sys/class/infiniband_mad) to build the per-function devices, sorted by PCI address. Done when the generated spec for a host with several ConnectX functions matches the JSON example, including Ethernet-only functions with an env entry and no device nodes.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Why

Kata runtime-rs annotates every ConnectX function it passes through to a VM as
cdi.k8s.io/vfio<N>: nvidia.com/nic=<index>, indices in guest PCI order
(src/runtime-rs/crates/resource/src/cdi_devices/mod.rs, row 0x15b3-0x020).
kata-agent resolves those names against /var/run/cdi before it creates the
container and fails the container after 100 s without a match. Nothing in the
guest writes a spec of that kind today, so any pod with a ConnectX PF or VF,
and the HGX Service VM holding the four NVSwitch management PFs, cannot start
its container under runtime-rs.

The GPU side has this covered: the guest init runs nvidia-ctk cdi generate
and the agent resolves nvidia.com/gpu=<index>. The NIC side needs the same
producer. --mode mofed is not it: it emits kind nvidia.com/gpu, one device
named all, every /dev/infiniband/uverbs* plus rdma_cm, no per-function
devices and no indices.

Ask

A discovery mode nic (name open) that emits, for every PCI function with
vendor 0x15b3 and base class 0x02 (Ethernet 0x0200 and InfiniBand
0x0207 alike, PF or VF), sorted by PCI address:

{
  "cdiVersion": "0.6.0",
  "kind": "nvidia.com/nic",
  "devices": [
    {"name": "0", "containerEdits": {"env": ["NVIDIA_NIC_0=0000:04:00.0"],
      "deviceNodes": [{"path": "/dev/infiniband/issm0"}, {"path": "/dev/infiniband/rdma_cm"},
                      {"path": "/dev/infiniband/umad0"}, {"path": "/dev/infiniband/uverbs0"}]}},
    {"name": "1", "containerEdits": {"env": ["NVIDIA_NIC_1=0000:05:00.0"], "deviceNodes": ["..."]}}
  ]
}
  • Device <index> is the function's position in PCI-address order; that is the
    order runtime-rs uses (sort_devices_by_guest_pcipath), and the same per-VM
    index contract the GPU spec already relies on.
  • Nodes come from the function's InfiniBand devices:
    /sys/bus/pci/devices/<bdf>/infiniband/<ibdev>, then every
    /sys/class/infiniband_verbs/uverbs*/ibdev and /sys/class/infiniband_mad/{umad,issm}*/ibdev
    equal to <ibdev>, plus /dev/infiniband/rdma_cm when present. A function
    without an InfiniBand device (Ethernet with RDMA off) still gets its device
    entry, so the index set stays complete; the env entry keeps the device valid
    (CDI rejects a device with no edits) and tells the container which function
    device <index> is.
  • No NVML requirement: the Service VM has no GPU. --mode mofed already runs
    without a driver ("Could not determine driver version ... ignoring").

Verified shape

A guest-side generator producing exactly this document was run on an HGX B300
with kata runtime-rs's row in place: the agent resolved four devices, injected
all of them, the container was Running 49 s after apply, and PID 1 carried
NVIDIA_NIC_0..3 with the guest addresses in root-port order. The document
above is that output. The generator is a bridge in the guest init (NVIDIA/nvrc#245)
and should be replaced by this toolkit mode.

Lenguaje dominante
Go
Estrellas
4.6k
Forks
612
Merge medio
2 d 9 h
PR fusionados (30 d)
39

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de NVIDIA/nvidia-container-toolkit

Todos los issues de NVIDIA/nvidia-container-toolkit

Issues similares

Más issues de Go

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.