Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

[Bug]: GPU Operator MPS config-manager cannot signal MPS daemon due to process-target mismatch

Open
#2,970 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
68/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
go, kubernetes

Research direction

Start with the MPS DaemonSet template and the config-manager implementation used by the v0.20.1 device-plugin image, comparing the mps-control-daemon command with PROCESS_TO_SIGNAL and its exact cmdline matching. Reproduce the profile switch with the supplied kubectl label command, then verify that config-manager signals the daemon successfully and the MPS DaemonSet returns to 2/2 Ready.

Written by the indexing model from the issue text.

Description

bug needs-triage

Describe the bug
With GPU Operator v26.7.0 and v26.7.1 (other versions not tested), the MPS control-daemon DaemonSet config-manager sidecar cannot reload the MPS control daemon after a device-plugin configuration change.

The MPS DaemonSet template starts the daemon using: command: [mps-control-daemon] but config-manager is configured to search for: PROCESS_TO_SIGNAL: /usr/bin/mps-control-daemon. The config-manager implementation requires an exact match against cmdline[0]. It cannot find the process, exits with code 1, and enters CrashLoopBackOff.

This leaves the MPS DaemonSet at 1/2 ready and the GPU Operator ClusterPolicy in notReady, even when the MPS main container is eventually serving shared-GPU resources.

To Reproduce

  1. Install GPU Operator v26.7.1 with a device-plugin ConfigMap containing two profiles: i.e. default without MPS (full GPU allocation) and one with MPS enabled. The values I used:

mig:
  strategy: none

devicePlugin:
  config:
    create: true
    name: nvidia-device-plugin-config
    default: fullGpu
    data:
      fullGpu: |-
        version: v1
        flags:
          migStrategy: none
      mps-profile: |-
        version: v1
        sharing:
          mps:
            renameByDefault: true
            resources:
              - name: nvidia.com/gpu
                replicas: 4
  mps:
    root: /run/nvidia/mps
  1. Allow daemons to start with default profile:
  2. Change the node to the MPS profile:
kubectl label node <gpu-node> nvidia.com/device-plugin.config=mps-profile --overwrite
  1. Check the MPS config-manager logs:
POD=$(kubectl get pods -n nvidia -l app=nvidia-device-plugin-mps-control-daemon -o jsonpath='{.items[0].metadata.name}')
kubectl logs -n nvidia "$POD" -c config-manager --tail=100

Actual result

Label change detected: nvidia.com/device-plugin.config=mps-profile
Updating to config: mps-profile
Successfully updated to config: mps-profile
Sending signal 'hangup' to '/usr/bin/mps-control-daemon'
error finding pid: no process found

The config-manager sidecar enters CrashLoopBackOff, the MPS DaemonSet remains 1/2, and ClusterPolicy stays notReady.

Expected behavior
The config-manager sends SIGHUP successfully, the MPS control daemon reloads the new profile, and the MPS DaemonSet remains 2/2 Ready.

Environment (please provide the following information):

  • GPU Operator Version: v26.7.1
  • OS: Ubuntu24.04
  • Kernel Version: 6.8.0-generic
  • Container Runtime Version: containerd 2.2.2-1ubuntu.24.04noble
  • Kubernetes Distro and Version: Kubernetes: v1.35.3

Information to attach (optional if deemed irrelevant)

  • kubernetes pods status: kubectl get pods -n OPERATOR_NAMESPACE

    # Other pods are running and completely healthy except the mps-control-daemon
    ...
    nvidia-device-plugin-mps-control-daemon-wqhfq                     1/2     CrashLoopBackOff   12 (106s ago)   39m
    ...
    
  • kubernetes daemonset status: kubectl get ds -n OPERATOR_NAMESPACE

    NAME                                      DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                                                          AGE
    nvidia-device-plugin-mps-control-daemon   1         1         0       1            0           nvidia.com/gpu.deploy.device-plugin=true,nvidia.com/mps.capable=true   3h36m
    
  • If a pod/ds is in an error state or pending state kubectl describe pod -n OPERATOR_NAMESPACE POD_NAME

     $ kubectl describe pod -n nvidia nvidia-device-plugin-mps-control-daemon-8vqms
     Name:                 nvidia-device-plugin-mps-control-daemon-8vqms
     Namespace:            nvidia
     Priority:             2000001000
     Priority Class Name:  system-node-critical
     Runtime Class Name:   nvidia
     Service Account:      nvidia-device-plugin
     Node:                 k8s-worker-gpu-0/172.22.4.22
     Start Time:           Fri, 25 Sep 2026 19:12:17 +0300
     Labels:               app=nvidia-device-plugin-mps-control-daemon
                           app.kubernetes.io/managed-by=gpu-operator
                           controller-revision-hash=956ddf497
                           helm.sh/chart=gpu-operator-v26.7.1
                           pod-template-generation=8
                           topology.kubernetes.io/region=RegionOne
                           topology.kubernetes.io/zone=nova
     Annotations:          <none>
     Status:               Running
     IP:                   10.244.221.17
     IPs:
       IP:           10.244.221.17
     Controlled By:  DaemonSet/nvidia-device-plugin-mps-control-daemon
     Init Containers:
       toolkit-validation:
         Container ID:  containerd://828c8ff9e644722a867b3df8d520a65dc64dea80a1be2770f9cc8419e4d3fa5c
         Image:         nvcr.io/nvidia/gpu-operator:v26.7.1
         Image ID:      nvcr.io/nvidia/gpu-operator@sha256:b844f9c033e748b8f9925f791f426a531ee8a855f04d4ccea365378634bcefe5
         Port:          <none>
         Host Port:     <none>
         Command:
           sh
           -c
         Args:
           until [ -f /run/nvidia/validations/toolkit-ready ] && { grep -q '^nvidia ' /proc/modules || [ -e /dev/dxg ]; }; do echo waiting for nvidia container stack to be setup; sleep 5; done
         State:          Terminated
           Reason:       Completed
           Exit Code:    0
           Started:      Fri, 25 Sep 2026 19:12:17 +0300
           Finished:     Fri, 25 Sep 2026 19:12:17 +0300
         Ready:          True
         Restart Count:  0
         Environment:    <none>
         Mounts:
           /run/nvidia from run-nvidia (rw)
           /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-z6xkw (ro)
       mps-control-daemon-mounts:
         Container ID:  containerd://6b72cb1e4350fd323d4a85bb8522412252f321be8868c488b923d5e4663c405b
         Image:         nvcr.io/nvidia/k8s-device-plugin:v0.20.1
         Image ID:      nvcr.io/nvidia/k8s-device-plugin@sha256:27c1b2553a690ca3d29889dcd61df97525d46f58272c0be8dc63c2b18f120075
         Port:          <none>
         Host Port:     <none>
         Command:
           mps-control-daemon
           mount-shm
         State:          Terminated
           Reason:       Completed
           Exit Code:    0
           Started:      Fri, 25 Sep 2026 19:12:18 +0300
           Finished:     Fri, 25 Sep 2026 19:12:18 +0300
         Ready:          True
         Restart Count:  0
         Environment:    <none>
         Mounts:
           /mps from mps-root (rw)
           /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-z6xkw (ro)
       config-manager-init:
         Container ID:  containerd://e81e9d6ce3c5fca148f959fa6558c661618d4520e301270f373f2314f40d5e82
         Image:         nvcr.io/nvidia/k8s-device-plugin:v0.20.1
         Image ID:      nvcr.io/nvidia/k8s-device-plugin@sha256:27c1b2553a690ca3d29889dcd61df97525d46f58272c0be8dc63c2b18f120075
         Port:          <none>
         Host Port:     <none>
         Command:
           config-manager
         State:          Terminated
           Reason:       Completed
           Exit Code:    0
           Started:      Fri, 25 Sep 2026 19:12:19 +0300
           Finished:     Fri, 25 Sep 2026 19:12:19 +0300
         Ready:          True
         Restart Count:  0
         Environment:
           ONESHOT:              true
           KUBECONFIG:
           NODE_NAME:             (v1:spec.nodeName)
           NODE_LABEL:           nvidia.com/device-plugin.config
           CONFIG_FILE_SRCDIR:   /available-configs
           CONFIG_FILE_DST:      /config/config.yaml
           DEFAULT_CONFIG:       fullGpu
           SEND_SIGNAL:          false
           SIGNAL:
           PROCESS_TO_SIGNAL:
           FALLBACK_STRATEGIES:  empty
         Mounts:
           /available-configs from nvidia-gpu-operator-device-plugin-config (rw)
           /config from config (rw)
           /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-z6xkw (ro)
     Containers:
       mps-control-daemon-ctr:
         Container ID:  containerd://7fe7a1e57c4cc978869e6da1cff72439c0241c236c3826550fb6b816d0409c20
         Image:         nvcr.io/nvidia/k8s-device-plugin:v0.20.1
         Image ID:      nvcr.io/nvidia/k8s-device-plugin@sha256:27c1b2553a690ca3d29889dcd61df97525d46f58272c0be8dc63c2b18f120075
         Port:          <none>
         Host Port:     <none>
         Command:
           mps-control-daemon
         State:          Running
           Started:      Fri, 25 Sep 2026 19:12:20 +0300
         Ready:          True
         Restart Count:  0
         Environment:
           NODE_NAME:                    (v1:spec.nodeName)
           NVIDIA_VISIBLE_DEVICES:      all
           NVIDIA_DRIVER_CAPABILITIES:  compute,utility
           CONFIG_FILE:                 /config/config.yaml
           MIG_STRATEGY:                none
         Mounts:
           /available-configs from nvidia-gpu-operator-device-plugin-config (rw)
           /config from config (rw)
           /dev/shm from mps-shm (rw)
           /mps from mps-root (rw)
           /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-z6xkw (ro)
       config-manager:
         Container ID:  containerd://83d50e253dd1b7c1393fe4188b6f20fc9f82786190234ee6128fb9ec25586141
         Image:         nvcr.io/nvidia/k8s-device-plugin:v0.20.1
         Image ID:      nvcr.io/nvidia/k8s-device-plugin@sha256:27c1b2553a690ca3d29889dcd61df97525d46f58272c0be8dc63c2b18f120075
         Port:          <none>
         Host Port:     <none>
         Command:
           config-manager
         State:          Terminated
           Reason:       Error
           Exit Code:    1
           Started:      Fri, 25 Sep 2026 19:12:34 +0300
           Finished:     Fri, 25 Sep 2026 19:12:34 +0300
         Last State:     Terminated
           Reason:       Error
           Exit Code:    1
           Started:      Fri, 25 Sep 2026 19:12:21 +0300
           Finished:     Fri, 25 Sep 2026 19:12:21 +0300
         Ready:          False
         Restart Count:  2
         Environment:
           ONESHOT:              false
           KUBECONFIG:
           NODE_NAME:             (v1:spec.nodeName)
           NODE_LABEL:           nvidia.com/device-plugin.config
           CONFIG_FILE_SRCDIR:   /available-configs
           CONFIG_FILE_DST:      /config/config.yaml
           DEFAULT_CONFIG:       fullGpu
           SEND_SIGNAL:          true
           SIGNAL:               1
           PROCESS_TO_SIGNAL:    /usr/bin/mps-control-daemon
           FALLBACK_STRATEGIES:  empty
         Mounts:
           /available-configs from nvidia-gpu-operator-device-plugin-config (rw)
           /config from config (rw)
           /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-z6xkw (ro)
     Conditions:
       Type                        Status
       PodReadyToStartContainers   True
       Initialized                 True
       Ready                       False
       ContainersReady             False
       PodScheduled                True
     Volumes:
       run-nvidia:
         Type:          HostPath (bare host directory volume)
         Path:          /run/nvidia
         HostPathType:  Directory
       mps-root:
         Type:          HostPath (bare host directory volume)
         Path:          /run/nvidia/mps
         HostPathType:  DirectoryOrCreate
       mps-shm:
         Type:          HostPath (bare host directory volume)
         Path:          /run/nvidia/mps/shm
         HostPathType:
       nvidia-gpu-operator-device-plugin-config:
         Type:      ConfigMap (a volume populated by a ConfigMap)
         Name:      nvidia-gpu-operator-device-plugin-config
         Optional:  false
       config:
         Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
         Medium:
         SizeLimit:  <unset>
       kube-api-access-z6xkw:
         Type:                    Projected (a volume that contains injected data from multiple sources)
         TokenExpirationSeconds:  3607
         ConfigMapName:           kube-root-ca.crt
         Optional:                false
         DownwardAPI:             true
     QoS Class:                   BestEffort
     Node-Selectors:              nvidia.com/gpu.deploy.device-plugin=true
                                  nvidia.com/mps.capable=true
     Tolerations:                 evo.company/nvidia-gpu:NoSchedule op=Exists
                                  node.kubernetes.io/disk-pressure:NoSchedule op=Exists
                                  node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                                  node.kubernetes.io/not-ready:NoExecute op=Exists
                                  node.kubernetes.io/pid-pressure:NoSchedule op=Exists
                                  node.kubernetes.io/unreachable:NoExecute op=Exists
                                  node.kubernetes.io/unschedulable:NoSchedule op=Exists
                                  nvidia.com/gpu:NoSchedule op=Exists
     Events:
       Type     Reason     Age                From               Message
       ----     ------     ----               ----               -------
       Normal   Scheduled  43s                default-scheduler  Successfully assigned nvidia/nvidia-device-plugin-mps-control-daemon-8vqms to k8s-worker-gpu-0
       Normal   Pulled     43s                kubelet            spec.initContainers{toolkit-validation}: Container image "nvcr.io/nvidia/gpu-operator:v26.7.1" already present on machine and can be accessed by the pod
       Normal   Created    43s                kubelet            spec.initContainers{toolkit-validation}: Container created
       Normal   Started    43s                kubelet            spec.initContainers{toolkit-validation}: Container started
       Normal   Pulled     42s                kubelet            spec.initContainers{mps-control-daemon-mounts}: Container image "nvcr.io/nvidia/k8s-device-plugin:v0.20.1" already present on machine and can be accessed by the pod
       Normal   Created    42s                kubelet            spec.initContainers{mps-control-daemon-mounts}: Container created
       Normal   Started    42s                kubelet            spec.initContainers{mps-control-daemon-mounts}: Container started
       Normal   Pulled     41s                kubelet            spec.initContainers{config-manager-init}: Container image "nvcr.io/nvidia/k8s-device-plugin:v0.20.1" already present on machine and can be accessed by the pod
       Normal   Created    41s                kubelet            spec.initContainers{config-manager-init}: Container created
       Normal   Started    41s                kubelet            spec.initContainers{config-manager-init}: Container started
       Normal   Pulled     40s                kubelet            spec.containers{mps-control-daemon-ctr}: Container image "nvcr.io/nvidia/k8s-device-plugin:v0.20.1" already present on machine and can be accessed by the pod
       Normal   Created    40s                kubelet            spec.containers{mps-control-daemon-ctr}: Container created
       Normal   Started    40s                kubelet            spec.containers{mps-control-daemon-ctr}: Container started
       Normal   Pulled     26s (x3 over 40s)  kubelet            spec.containers{config-manager}: Container image "nvcr.io/nvidia/k8s-device-plugin:v0.20.1" already present on machine and can be accessed by the pod
       Normal   Created    26s (x3 over 40s)  kubelet            spec.containers{config-manager}: Container created
       Normal   Started    26s (x3 over 40s)  kubelet            spec.containers{config-manager}: Container started
       Warning  BackOff    26s (x2 over 38s)  kubelet            spec.containers{config-manager}: Back-off restarting failed container config-manager in pod nvidia-device-plugin-mps-control-daemon-8vqms_nvidia(2956c3ac-7f02-4752-a362-2950f79074be)
    
  • If a pod/ds is in an error state or pending state kubectl logs -n OPERATOR_NAMESPACE POD_NAME --all-containers

     kubectl logs -n nvidia nvidia-device-plugin-mps-control-daemon-8vqms --all-containers
     I0925 16:12:18.620956   72937 main.go:80] "NVIDIA MPS Control Daemon" version=<
     	487d7a93
     	commit: 487d7a93894b0f3ad08097169593972eb3ca8206
      >
     W0925 16:12:19.627450   73059 client_config.go:683] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
     I0925 16:12:19.627665   73059 main.go:248] Waiting for change to 'nvidia.com/device-plugin.config' label
     I0925 16:12:19.627672   73059 main.go:250] Label change detected: nvidia.com/device-plugin.config=
     I0925 16:12:19.627697   73059 main.go:362] No value set. Selecting default name: fullGpu
     I0925 16:12:19.627701   73059 main.go:306] Updating to config: fullGpu
     I0925 16:12:19.627733   73059 main.go:321] Successfully updated to config: fullGpu
     I0925 16:12:20.651387   73188 main.go:80] "NVIDIA MPS Control Daemon" version=<
     	487d7a93
     	commit: 487d7a93894b0f3ad08097169593972eb3ca8206
      >
     I0925 16:12:20.651458   73188 main.go:109] Starting OS watcher.
     I0925 16:12:20.651602   73188 main.go:123] Starting Daemons.
     I0925 16:12:20.651616   73188 main.go:166] Loading configuration.
     I0925 16:12:20.651802   73188 main.go:181] Updating config with default resource matching patterns.
     I0925 16:12:20.651860   73188 main.go:192]
     Running with config:
     {
       "version": "v1",
       "flags": {
         "migStrategy": "none",
         "failOnInitError": null,
         "gdrcopyEnabled": null,
         "gdsEnabled": null,
         "mofedEnabled": null,
         "useNodeFeatureAPI": null,
         "deviceDiscoveryStrategy": null,
         "plugin": {
           "passDeviceSpecs": null,
           "deviceListStrategy": null,
           "deviceIDStrategy": null,
           "cdiAnnotationPrefix": null,
           "nvidiaCTKPath": null,
           "containerDriverRoot": null,
           "sharedDevicesAllocationPolicy": null
         }
       },
       "resources": {
         "gpus": [
           {
             "pattern": "*",
             "name": "nvidia.com/gpu"
           }
         ]
       },
       "sharing": {
         "timeSlicing": {}
       },
       "imex": {}
     }
     I0925 16:12:20.651866   73188 main.go:196] Retrieving MPS daemons.
     I0925 16:12:20.651875   73188 manager.go:66] "Sharing strategy is not MPS; skipping MPS manager creation" strategy="none"
     I0925 16:12:20.651887   73188 main.go:205] No devices are configured for MPS sharing; Waiting indefinitely.
     W0925 16:15:10.260092   74999 client_config.go:683] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
     I0925 16:15:10.260297   74999 main.go:248] Waiting for change to 'nvidia.com/device-plugin.config' label
     I0925 16:15:10.260304   74999 main.go:250] Label change detected: nvidia.com/device-plugin.config=
     I0925 16:15:10.260326   74999 main.go:362] No value set. Selecting default name: fullGpu
     I0925 16:15:10.260331   74999 main.go:306] Updating to config: fullGpu
     I0925 16:15:10.260374   74999 main.go:321] Successfully updated to config: fullGpu
     I0925 16:15:10.260378   74999 main.go:325] Sending signal 'hangup' to '/usr/bin/mps-control-daemon'
     E0925 16:15:10.264400   74999 main.go:210] error finding pid: no process found
    
  • Output from running nvidia-smi from the driver container: kubectl exec DRIVER_POD_NAME -n OPERATOR_NAMESPACE -c nvidia-driver-ctr -- nvidia-smi

    Fri Sep 25 16:20:15 2026
    +-----------------------------------------------------------------------------------------+
    | NVIDIA-SMI 595.91.07              Driver Version: 595.91.07      CUDA Version: 13.2     |
    +-----------------------------------------+------------------------+----------------------+
    | GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
    | Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
    |                                         |                        |               MIG M. |
    |=========================================+========================+======================|
    |   0  NVIDIA GeForce RTX 5080        On  |   00000000:00:06.0 Off |                  N/A |
    |  0%   39C    P8             16W /  360W |       1MiB /  16303MiB |      0%      Default |
    |                                         |                        |                  N/A |
    +-----------------------------------------+------------------------+----------------------+
    |   1  NVIDIA GeForce RTX 5080        On  |   00000000:00:07.0 Off |                  N/A |
    |  0%   38C    P8             11W /  360W |       1MiB /  16303MiB |      0%      Default |
    |                                         |                        |                  N/A |
    +-----------------------------------------+------------------------+----------------------+
    |   2  NVIDIA GeForce RTX 5080        On  |   00000000:00:08.0 Off |                  N/A |
    |  0%   39C    P8             15W /  360W |       1MiB /  16303MiB |      0%      Default |
    |                                         |                        |                  N/A |
    +-----------------------------------------+------------------------+----------------------+
    |   3  NVIDIA GeForce RTX 5080        On  |   00000000:00:09.0 Off |                  N/A |
    |  0%   38C    P8             12W /  360W |       1MiB /  16303MiB |      0%      Default |
    |                                         |                        |                  N/A |
    +-----------------------------------------+------------------------+----------------------+
    
    +-----------------------------------------------------------------------------------------+
    | Processes:                                                                              |
    |  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
    |        ID   ID                                                               Usage      |
    |=========================================================================================|
    |  No running processes found                                                             |
    +-----------------------------------------------------------------------------------------+
    

Details
Using AI assistant to debug, I managed to find the following cause:

GPU Operator v26.7.1 defines:

  # assets/state-mps-control-daemon/0400_daemonset.yaml
  command: [mps-control-daemon]

and:

  - name: SEND_SIGNAL
    value: "true"
  - name: SIGNAL
    value: "1"
  - name: PROCESS_TO_SIGNAL
    value: "/usr/bin/mps-control-daemon"

Sources:

config-manager uses an exact command-line match

In k8s-device-plugin v0.20.1, config-manager searches processes and compares the first command-line element exactly:

  if cmdline[0] == f.ProcessToSignal {
      return p.PID, nil
  }

Source:

The same file returns the observed error if no matching process is found:

  return fmt.Errorf("error finding pid: %v", err)

Source:

Dominant language
Go
Stars
2.9k
Forks
552
Avg merge
1d 21h
Merged PRs (30d)
78

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from NVIDIA/gpu-operator

All issues in NVIDIA/gpu-operator

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.