[Bug]: GPU Operator MPS config-manager cannot signal MPS daemon due to process-target mismatch
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 68/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Active
- Tech stack
- go, kubernetes
- Domain
- devops, infrastructure
Research direction
Start with the MPS DaemonSet template and the config-manager implementation used by the v0.20.1 device-plugin image, comparing the mps-control-daemon command with PROCESS_TO_SIGNAL and its exact cmdline matching. Reproduce the profile switch with the supplied kubectl label command, then verify that config-manager signals the daemon successfully and the MPS DaemonSet returns to 2/2 Ready.
Written by the indexing model from the issue text.
Description
Describe the bug
With GPU Operator v26.7.0 and v26.7.1 (other versions not tested), the MPS control-daemon DaemonSet config-manager sidecar cannot reload the MPS control daemon after a device-plugin configuration change.
The MPS DaemonSet template starts the daemon using: command: [mps-control-daemon] but config-manager is configured to search for: PROCESS_TO_SIGNAL: /usr/bin/mps-control-daemon. The config-manager implementation requires an exact match against cmdline[0]. It cannot find the process, exits with code 1, and enters CrashLoopBackOff.
This leaves the MPS DaemonSet at 1/2 ready and the GPU Operator ClusterPolicy in notReady, even when the MPS main container is eventually serving shared-GPU resources.
To Reproduce
- Install GPU Operator v26.7.1 with a device-plugin ConfigMap containing two profiles: i.e. default without MPS (full GPU allocation) and one with MPS enabled. The values I used:
mig:
strategy: none
devicePlugin:
config:
create: true
name: nvidia-device-plugin-config
default: fullGpu
data:
fullGpu: |-
version: v1
flags:
migStrategy: none
mps-profile: |-
version: v1
sharing:
mps:
renameByDefault: true
resources:
- name: nvidia.com/gpu
replicas: 4
mps:
root: /run/nvidia/mps
- Allow daemons to start with default profile:
- Change the node to the MPS profile:
kubectl label node <gpu-node> nvidia.com/device-plugin.config=mps-profile --overwrite
- Check the MPS config-manager logs:
POD=$(kubectl get pods -n nvidia -l app=nvidia-device-plugin-mps-control-daemon -o jsonpath='{.items[0].metadata.name}')
kubectl logs -n nvidia "$POD" -c config-manager --tail=100
Actual result
Label change detected: nvidia.com/device-plugin.config=mps-profile
Updating to config: mps-profile
Successfully updated to config: mps-profile
Sending signal 'hangup' to '/usr/bin/mps-control-daemon'
error finding pid: no process found
The config-manager sidecar enters CrashLoopBackOff, the MPS DaemonSet remains 1/2, and ClusterPolicy stays notReady.
Expected behavior
The config-manager sends SIGHUP successfully, the MPS control daemon reloads the new profile, and the MPS DaemonSet remains 2/2 Ready.
Environment (please provide the following information):
- GPU Operator Version: v26.7.1
- OS: Ubuntu24.04
- Kernel Version: 6.8.0-generic
- Container Runtime Version: containerd 2.2.2-1
ubuntu.24.04noble - Kubernetes Distro and Version: Kubernetes: v1.35.3
Information to attach (optional if deemed irrelevant)
-
kubernetes pods status:
kubectl get pods -n OPERATOR_NAMESPACE# Other pods are running and completely healthy except the mps-control-daemon ... nvidia-device-plugin-mps-control-daemon-wqhfq 1/2 CrashLoopBackOff 12 (106s ago) 39m ... -
kubernetes daemonset status:
kubectl get ds -n OPERATOR_NAMESPACENAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE nvidia-device-plugin-mps-control-daemon 1 1 0 1 0 nvidia.com/gpu.deploy.device-plugin=true,nvidia.com/mps.capable=true 3h36m -
If a pod/ds is in an error state or pending state
kubectl describe pod -n OPERATOR_NAMESPACE POD_NAME$ kubectl describe pod -n nvidia nvidia-device-plugin-mps-control-daemon-8vqms Name: nvidia-device-plugin-mps-control-daemon-8vqms Namespace: nvidia Priority: 2000001000 Priority Class Name: system-node-critical Runtime Class Name: nvidia Service Account: nvidia-device-plugin Node: k8s-worker-gpu-0/172.22.4.22 Start Time: Fri, 25 Sep 2026 19:12:17 +0300 Labels: app=nvidia-device-plugin-mps-control-daemon app.kubernetes.io/managed-by=gpu-operator controller-revision-hash=956ddf497 helm.sh/chart=gpu-operator-v26.7.1 pod-template-generation=8 topology.kubernetes.io/region=RegionOne topology.kubernetes.io/zone=nova Annotations: <none> Status: Running IP: 10.244.221.17 IPs: IP: 10.244.221.17 Controlled By: DaemonSet/nvidia-device-plugin-mps-control-daemon Init Containers: toolkit-validation: Container ID: containerd://828c8ff9e644722a867b3df8d520a65dc64dea80a1be2770f9cc8419e4d3fa5c Image: nvcr.io/nvidia/gpu-operator:v26.7.1 Image ID: nvcr.io/nvidia/gpu-operator@sha256:b844f9c033e748b8f9925f791f426a531ee8a855f04d4ccea365378634bcefe5 Port: <none> Host Port: <none> Command: sh -c Args: until [ -f /run/nvidia/validations/toolkit-ready ] && { grep -q '^nvidia ' /proc/modules || [ -e /dev/dxg ]; }; do echo waiting for nvidia container stack to be setup; sleep 5; done State: Terminated Reason: Completed Exit Code: 0 Started: Fri, 25 Sep 2026 19:12:17 +0300 Finished: Fri, 25 Sep 2026 19:12:17 +0300 Ready: True Restart Count: 0 Environment: <none> Mounts: /run/nvidia from run-nvidia (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-z6xkw (ro) mps-control-daemon-mounts: Container ID: containerd://6b72cb1e4350fd323d4a85bb8522412252f321be8868c488b923d5e4663c405b Image: nvcr.io/nvidia/k8s-device-plugin:v0.20.1 Image ID: nvcr.io/nvidia/k8s-device-plugin@sha256:27c1b2553a690ca3d29889dcd61df97525d46f58272c0be8dc63c2b18f120075 Port: <none> Host Port: <none> Command: mps-control-daemon mount-shm State: Terminated Reason: Completed Exit Code: 0 Started: Fri, 25 Sep 2026 19:12:18 +0300 Finished: Fri, 25 Sep 2026 19:12:18 +0300 Ready: True Restart Count: 0 Environment: <none> Mounts: /mps from mps-root (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-z6xkw (ro) config-manager-init: Container ID: containerd://e81e9d6ce3c5fca148f959fa6558c661618d4520e301270f373f2314f40d5e82 Image: nvcr.io/nvidia/k8s-device-plugin:v0.20.1 Image ID: nvcr.io/nvidia/k8s-device-plugin@sha256:27c1b2553a690ca3d29889dcd61df97525d46f58272c0be8dc63c2b18f120075 Port: <none> Host Port: <none> Command: config-manager State: Terminated Reason: Completed Exit Code: 0 Started: Fri, 25 Sep 2026 19:12:19 +0300 Finished: Fri, 25 Sep 2026 19:12:19 +0300 Ready: True Restart Count: 0 Environment: ONESHOT: true KUBECONFIG: NODE_NAME: (v1:spec.nodeName) NODE_LABEL: nvidia.com/device-plugin.config CONFIG_FILE_SRCDIR: /available-configs CONFIG_FILE_DST: /config/config.yaml DEFAULT_CONFIG: fullGpu SEND_SIGNAL: false SIGNAL: PROCESS_TO_SIGNAL: FALLBACK_STRATEGIES: empty Mounts: /available-configs from nvidia-gpu-operator-device-plugin-config (rw) /config from config (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-z6xkw (ro) Containers: mps-control-daemon-ctr: Container ID: containerd://7fe7a1e57c4cc978869e6da1cff72439c0241c236c3826550fb6b816d0409c20 Image: nvcr.io/nvidia/k8s-device-plugin:v0.20.1 Image ID: nvcr.io/nvidia/k8s-device-plugin@sha256:27c1b2553a690ca3d29889dcd61df97525d46f58272c0be8dc63c2b18f120075 Port: <none> Host Port: <none> Command: mps-control-daemon State: Running Started: Fri, 25 Sep 2026 19:12:20 +0300 Ready: True Restart Count: 0 Environment: NODE_NAME: (v1:spec.nodeName) NVIDIA_VISIBLE_DEVICES: all NVIDIA_DRIVER_CAPABILITIES: compute,utility CONFIG_FILE: /config/config.yaml MIG_STRATEGY: none Mounts: /available-configs from nvidia-gpu-operator-device-plugin-config (rw) /config from config (rw) /dev/shm from mps-shm (rw) /mps from mps-root (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-z6xkw (ro) config-manager: Container ID: containerd://83d50e253dd1b7c1393fe4188b6f20fc9f82786190234ee6128fb9ec25586141 Image: nvcr.io/nvidia/k8s-device-plugin:v0.20.1 Image ID: nvcr.io/nvidia/k8s-device-plugin@sha256:27c1b2553a690ca3d29889dcd61df97525d46f58272c0be8dc63c2b18f120075 Port: <none> Host Port: <none> Command: config-manager State: Terminated Reason: Error Exit Code: 1 Started: Fri, 25 Sep 2026 19:12:34 +0300 Finished: Fri, 25 Sep 2026 19:12:34 +0300 Last State: Terminated Reason: Error Exit Code: 1 Started: Fri, 25 Sep 2026 19:12:21 +0300 Finished: Fri, 25 Sep 2026 19:12:21 +0300 Ready: False Restart Count: 2 Environment: ONESHOT: false KUBECONFIG: NODE_NAME: (v1:spec.nodeName) NODE_LABEL: nvidia.com/device-plugin.config CONFIG_FILE_SRCDIR: /available-configs CONFIG_FILE_DST: /config/config.yaml DEFAULT_CONFIG: fullGpu SEND_SIGNAL: true SIGNAL: 1 PROCESS_TO_SIGNAL: /usr/bin/mps-control-daemon FALLBACK_STRATEGIES: empty Mounts: /available-configs from nvidia-gpu-operator-device-plugin-config (rw) /config from config (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-z6xkw (ro) Conditions: Type Status PodReadyToStartContainers True Initialized True Ready False ContainersReady False PodScheduled True Volumes: run-nvidia: Type: HostPath (bare host directory volume) Path: /run/nvidia HostPathType: Directory mps-root: Type: HostPath (bare host directory volume) Path: /run/nvidia/mps HostPathType: DirectoryOrCreate mps-shm: Type: HostPath (bare host directory volume) Path: /run/nvidia/mps/shm HostPathType: nvidia-gpu-operator-device-plugin-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: nvidia-gpu-operator-device-plugin-config Optional: false config: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: SizeLimit: <unset> kube-api-access-z6xkw: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt Optional: false DownwardAPI: true QoS Class: BestEffort Node-Selectors: nvidia.com/gpu.deploy.device-plugin=true nvidia.com/mps.capable=true Tolerations: evo.company/nvidia-gpu:NoSchedule op=Exists node.kubernetes.io/disk-pressure:NoSchedule op=Exists node.kubernetes.io/memory-pressure:NoSchedule op=Exists node.kubernetes.io/not-ready:NoExecute op=Exists node.kubernetes.io/pid-pressure:NoSchedule op=Exists node.kubernetes.io/unreachable:NoExecute op=Exists node.kubernetes.io/unschedulable:NoSchedule op=Exists nvidia.com/gpu:NoSchedule op=Exists Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 43s default-scheduler Successfully assigned nvidia/nvidia-device-plugin-mps-control-daemon-8vqms to k8s-worker-gpu-0 Normal Pulled 43s kubelet spec.initContainers{toolkit-validation}: Container image "nvcr.io/nvidia/gpu-operator:v26.7.1" already present on machine and can be accessed by the pod Normal Created 43s kubelet spec.initContainers{toolkit-validation}: Container created Normal Started 43s kubelet spec.initContainers{toolkit-validation}: Container started Normal Pulled 42s kubelet spec.initContainers{mps-control-daemon-mounts}: Container image "nvcr.io/nvidia/k8s-device-plugin:v0.20.1" already present on machine and can be accessed by the pod Normal Created 42s kubelet spec.initContainers{mps-control-daemon-mounts}: Container created Normal Started 42s kubelet spec.initContainers{mps-control-daemon-mounts}: Container started Normal Pulled 41s kubelet spec.initContainers{config-manager-init}: Container image "nvcr.io/nvidia/k8s-device-plugin:v0.20.1" already present on machine and can be accessed by the pod Normal Created 41s kubelet spec.initContainers{config-manager-init}: Container created Normal Started 41s kubelet spec.initContainers{config-manager-init}: Container started Normal Pulled 40s kubelet spec.containers{mps-control-daemon-ctr}: Container image "nvcr.io/nvidia/k8s-device-plugin:v0.20.1" already present on machine and can be accessed by the pod Normal Created 40s kubelet spec.containers{mps-control-daemon-ctr}: Container created Normal Started 40s kubelet spec.containers{mps-control-daemon-ctr}: Container started Normal Pulled 26s (x3 over 40s) kubelet spec.containers{config-manager}: Container image "nvcr.io/nvidia/k8s-device-plugin:v0.20.1" already present on machine and can be accessed by the pod Normal Created 26s (x3 over 40s) kubelet spec.containers{config-manager}: Container created Normal Started 26s (x3 over 40s) kubelet spec.containers{config-manager}: Container started Warning BackOff 26s (x2 over 38s) kubelet spec.containers{config-manager}: Back-off restarting failed container config-manager in pod nvidia-device-plugin-mps-control-daemon-8vqms_nvidia(2956c3ac-7f02-4752-a362-2950f79074be) -
If a pod/ds is in an error state or pending state
kubectl logs -n OPERATOR_NAMESPACE POD_NAME --all-containerskubectl logs -n nvidia nvidia-device-plugin-mps-control-daemon-8vqms --all-containers I0925 16:12:18.620956 72937 main.go:80] "NVIDIA MPS Control Daemon" version=< 487d7a93 commit: 487d7a93894b0f3ad08097169593972eb3ca8206 > W0925 16:12:19.627450 73059 client_config.go:683] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work. I0925 16:12:19.627665 73059 main.go:248] Waiting for change to 'nvidia.com/device-plugin.config' label I0925 16:12:19.627672 73059 main.go:250] Label change detected: nvidia.com/device-plugin.config= I0925 16:12:19.627697 73059 main.go:362] No value set. Selecting default name: fullGpu I0925 16:12:19.627701 73059 main.go:306] Updating to config: fullGpu I0925 16:12:19.627733 73059 main.go:321] Successfully updated to config: fullGpu I0925 16:12:20.651387 73188 main.go:80] "NVIDIA MPS Control Daemon" version=< 487d7a93 commit: 487d7a93894b0f3ad08097169593972eb3ca8206 > I0925 16:12:20.651458 73188 main.go:109] Starting OS watcher. I0925 16:12:20.651602 73188 main.go:123] Starting Daemons. I0925 16:12:20.651616 73188 main.go:166] Loading configuration. I0925 16:12:20.651802 73188 main.go:181] Updating config with default resource matching patterns. I0925 16:12:20.651860 73188 main.go:192] Running with config: { "version": "v1", "flags": { "migStrategy": "none", "failOnInitError": null, "gdrcopyEnabled": null, "gdsEnabled": null, "mofedEnabled": null, "useNodeFeatureAPI": null, "deviceDiscoveryStrategy": null, "plugin": { "passDeviceSpecs": null, "deviceListStrategy": null, "deviceIDStrategy": null, "cdiAnnotationPrefix": null, "nvidiaCTKPath": null, "containerDriverRoot": null, "sharedDevicesAllocationPolicy": null } }, "resources": { "gpus": [ { "pattern": "*", "name": "nvidia.com/gpu" } ] }, "sharing": { "timeSlicing": {} }, "imex": {} } I0925 16:12:20.651866 73188 main.go:196] Retrieving MPS daemons. I0925 16:12:20.651875 73188 manager.go:66] "Sharing strategy is not MPS; skipping MPS manager creation" strategy="none" I0925 16:12:20.651887 73188 main.go:205] No devices are configured for MPS sharing; Waiting indefinitely. W0925 16:15:10.260092 74999 client_config.go:683] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work. I0925 16:15:10.260297 74999 main.go:248] Waiting for change to 'nvidia.com/device-plugin.config' label I0925 16:15:10.260304 74999 main.go:250] Label change detected: nvidia.com/device-plugin.config= I0925 16:15:10.260326 74999 main.go:362] No value set. Selecting default name: fullGpu I0925 16:15:10.260331 74999 main.go:306] Updating to config: fullGpu I0925 16:15:10.260374 74999 main.go:321] Successfully updated to config: fullGpu I0925 16:15:10.260378 74999 main.go:325] Sending signal 'hangup' to '/usr/bin/mps-control-daemon' E0925 16:15:10.264400 74999 main.go:210] error finding pid: no process found -
Output from running
nvidia-smifrom the driver container:kubectl exec DRIVER_POD_NAME -n OPERATOR_NAMESPACE -c nvidia-driver-ctr -- nvidia-smiFri Sep 25 16:20:15 2026 +-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 595.91.07 Driver Version: 595.91.07 CUDA Version: 13.2 | +-----------------------------------------+------------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+========================+======================| | 0 NVIDIA GeForce RTX 5080 On | 00000000:00:06.0 Off | N/A | | 0% 39C P8 16W / 360W | 1MiB / 16303MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ | 1 NVIDIA GeForce RTX 5080 On | 00000000:00:07.0 Off | N/A | | 0% 38C P8 11W / 360W | 1MiB / 16303MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ | 2 NVIDIA GeForce RTX 5080 On | 00000000:00:08.0 Off | N/A | | 0% 39C P8 15W / 360W | 1MiB / 16303MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ | 3 NVIDIA GeForce RTX 5080 On | 00000000:00:09.0 Off | N/A | | 0% 38C P8 12W / 360W | 1MiB / 16303MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ +-----------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=========================================================================================| | No running processes found | +-----------------------------------------------------------------------------------------+
Details
Using AI assistant to debug, I managed to find the following cause:
GPU Operator v26.7.1 defines:
# assets/state-mps-control-daemon/0400_daemonset.yaml
command: [mps-control-daemon]
and:
- name: SEND_SIGNAL
value: "true"
- name: SIGNAL
value: "1"
- name: PROCESS_TO_SIGNAL
value: "/usr/bin/mps-control-daemon"
Sources:
- https://github.com/NVIDIA/gpu-operator/blob/v26.7.1/assets/state-mps-control-daemon/0400_daemonset.yaml#L78
- https://github.com/NVIDIA/gpu-operator/blob/v26.7.1/assets/state-mps-control-daemon/0400_daemonset.yaml#L118-L123
config-manager uses an exact command-line match
In k8s-device-plugin v0.20.1, config-manager searches processes and compares the first command-line element exactly:
if cmdline[0] == f.ProcessToSignal {
return p.PID, nil
}
Source:
The same file returns the observed error if no matching process is found:
return fmt.Errorf("error finding pid: %v", err)
Source:
- Dominant language
- Go
- Stars
- 2.9k
- Forks
- 552
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 78
Getting set up
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from NVIDIA/gpu-operator
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
NVIDIA/gpu-operator#2968 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
NVIDIA/gpu-operator#2955 ·
Maintainers usually reply within 1 day
-
lifecycle/stale question
Difficulty 2/5 1-3 hours Newbie friendliness 64/100
NVIDIA/gpu-operator#2280 · 2 comments · 1 reaction ·
Maintainers usually reply within 1 day
-
Difficulty 4/5 3-5 days Newbie friendliness 68/100
NVIDIA/gpu-operator#2957 ·
Maintainers usually reply within 1 day
-
feature lifecycle/frozen needs-triage
Difficulty 3/5 1-2 days Newbie friendliness 65/100
NVIDIA/gpu-operator#2938 ·
Maintainers usually reply within 1 day
All issues in NVIDIA/gpu-operator
Similar issues
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
volodya-lombrozo/aidy#333 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
rossoctl/context-guru#314 ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 86/100
OwO-Network/DLX#236 · 1 comment · 1 reaction ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100