[Bug]: a partially specified NVIDIADriver startupProbe renders zeros instead of operator defaults

Open
  • Is something wrong?
#3,132 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

@blacksoxx is already working on this.

Since Oct 28, 2026.

  • #3133 by @blacksoxx — open

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
67/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
go, kubernetes
Domain
backend, devops

Research direction

Start in internal/state/driver.go at getDriverSpec and compare its startup probe handling with setContainerProbe in controllers/object_controls.go. Run the internal/state unit tests and inspect the rendered DaemonSet in manifests/state-driver/0500_daemonset.yaml. Done means a partially specified probe preserves user-set values and uses operator defaults for the remaining fields.

Written by the indexing model from the issue text.

Description

Describe the bug

When an NVIDIADriver CR sets spec.startupProbe but leaves some fields unset, the GPU Operator renders explicit zeros for those fields into the driver DaemonSet, so the driver startup window collapses to the generic Kubernetes probe defaults instead of the operator's intended values.

getDriverSpec (internal/state/driver.go) applies getDefaultStartupProbe only when spec.StartupProbe == nil:

if spec.StartupProbe == nil {
    spec.StartupProbe = getDefaultStartupProbe(spec)
}

If the CR specifies the probe partially (for example startupProbe: {initialDelaySeconds: 120}), the unset fields stay zero, and manifests/state-driver/0500_daemonset.yaml renders failureThreshold: 0, periodSeconds: 0, timeoutSeconds: 0, successThreshold: 0. The API server then substitutes the generic defaults (failureThreshold 3, periodSeconds 10, timeoutSeconds 1) rather than the operator defaults (failureThreshold 120, timeoutSeconds 60). A driver that compiles for more than ~30s, or whose nvidia-smi probe takes longer than 1s, is then killed and restarted in a loop.

The ClusterPolicy path does not have this problem: setContainerProbe (controllers/object_controls.go) overlays only the non-zero user fields on top of the defaults. The two paths disagree.

To Reproduce

Apply an NVIDIADriver CR whose spec.startupProbe sets only initialDelaySeconds (any single field), and inspect the rendered driver DaemonSet: the other probe fields are 0.

Reproduced with a unit test over getDriverSpec: a CR with StartupProbe{InitialDelaySeconds: 120} yields failureThreshold 0 / periodSeconds 0 / timeoutSeconds 0 / successThreshold 0 instead of the operator defaults.

Expected behavior

A partially specified startupProbe keeps the fields the user set and falls back to the operator defaults for the rest, as the ClusterPolicy path already does.

Environment (please provide the following information):

  • GPU Operator Version: present in v26.7.2 and on main (internal/state/driver.go has no overlay for a partial probe); reproduced on main
  • OS: N/A for the reproduction (unit test, macOS arm64, Go 1.26)
  • Kernel Version: N/A (not run on a cluster)
  • Container Runtime Version: N/A (not run on a cluster)
  • Kubernetes Distro and Version: N/A (not run on a cluster; reproduced with the internal/state unit tests)

A fix is in the linked PR.

Dominant language
Go
Stars
2.9k
Forks
587
Avg merge
1d 22h
Merged PRs (30d)
63

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from NVIDIA/gpu-operator

All issues in NVIDIA/gpu-operator

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.