Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

[Feature]: Start Fabric Manager in vGPU multitenancy mode (NVFM_CONFIG_FABRIC_MODE=2) on the vGPU Manager path

Open
  • Is something wrong?
#2,923 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
kubernetes
Domain
devops, infrastructure

Research direction

Start by comparing the vgpu-manager/ path with the gpu-driver-container entrypoint handling of NVFM_CONFIG_* and the existing vgpuManager.env and /run/nvidia-fabricmanager mount. Resolve whether NVSwitch vGPU support belongs on this path and whether the GPU reset applies to mode 2 before implementing anything. Done means the agreed path installs and starts Fabric Manager with NVFM_CONFIG_FABRIC_MODE=2.

Written by the indexing model from the issue text.

Description

@shengnuo as requested in NVIDIA/gpu-operator#2552, here is the vGPU side of the same problem.

I would like the vGPU Manager images to ship Fabric Manager and start it on NVSwitch hosts, with the mode set through the mechanism NVIDIA/gpu-driver-container#893 already added, so that the mode comes from NVFM_CONFIG_FABRIC_MODE=2 rather than from a second knob invented for vGPU.

The Fabric Manager User Guide says the vGPU host image must have the FM package and the FM SDK package installed, and "To support vGPU virtualization, start the FM service in vGPU virtualization mode by setting the FABRIC_MODE=2 FM config item". Partition activation is always required before starting a vGPU VM, even for VMs that use only one vGPU. On an 8-GPU HGX H200 host, with FM left in the default bare metal mode, a whole-card vGPU guest could not initialize CUDA: cuInit returned 802 and the guest reported fabric status "Insufficient Permissions". Mode 2 plus per-VM partition activation fixed it. The activation itself lives in NVIDIA/kubevirt-gpu-device-plugin#193.

In gpu-driver-container the nvidia-driver entrypoint collects NVFM_CONFIG_* from the environment and writes those keys into fabricmanager.cfg before starting FM. None of that reaches the vGPU Manager images. I find no Fabric Manager handling under vgpu-manager/, so those images neither install nor start it. In gpu-operator neither FABRIC_MODE nor NVFM_CONFIG_ appears anywhere, though vgpuManager.env already reaches the container and the vGPU host manager daemonset rendered from an NVIDIADriver CR already mounts /run/nvidia-fabricmanager from the host.

So on an operator-managed NVSwitch vGPU node, FM still has to be installed and configured out of band.

Two questions before any code:

  1. Is this wanted on the vGPU Manager path at all, given that the fabric work is heading into the DRA driver? If vGPU on NVSwitch is meant to be served there instead, I would rather hear it now.
  2. If it is wanted, does the GPU reset added in NVIDIA/gpu-driver-container#909 for NVLink5 systems apply to mode 2? Today it triggers on the literal value 1.

I have two drafts from before #893 that use their own FABRIC_MODE variable, NVIDIA/gpu-operator#2602 and NVIDIA/gpu-driver-container#854. If the answer to the first question is yes, I will rebase both onto NVFM_CONFIG_FABRIC_MODE and open them for review.

Dominant language
Go
Stars
2.9k
Forks
552
Avg merge
1d 21h
Merged PRs (30d)
78

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from NVIDIA/gpu-operator

All issues in NVIDIA/gpu-operator

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.