[Feature]: Ignore already cordoned nodes during driver upgrades
Maintainers usually reply within 1 day
@JunAr7112 is already working on this.
Since Sep 10, 2026.
Assessment
This issue has not been assessed yet.
Description
Requestor: @kvalliyurnatt
Summary
Ignore nodes that are already cordoned (not by any of the GPU operator components) during upgrades from upgrade controller/k8s-driver-manager
Motivation
If there is a node that has already been cordoned (not by any of the GPU operator components), then that means the node is not accepting any workloads at the moment, and there is no guarantee of the state that node is in, it could be down or in an unresponsive state. So it feels like the upgrade controller/k8s-driver-manager should ignore nodes that are already cordoned and not take any action on such nodes until they are uncordoned.
Proposal
We will need a way differentiate nodes that are cordoned by the GPU operator components, for which we could add an annotation on the nodes when the upgrade controller/k8s-driver-manager cordons a node(I believe the upgrade controller already does this) and then uncordon nodes based on that annotation. Then the upgrade controller/k8s-driver-manager can ignore nodes that are in a cordoned state but don't have the annotation on them.
- Dominant language
- Go
- Stars
- 2.9k
- Forks
- 552
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 78
Getting set up
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from NVIDIA/gpu-operator
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
NVIDIA/gpu-operator#2968 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
NVIDIA/gpu-operator#2955 ·
Maintainers usually reply within 1 day
-
lifecycle/stale question
Difficulty 2/5 1-3 hours Newbie friendliness 64/100
NVIDIA/gpu-operator#2280 · 2 comments · 1 reaction ·
Maintainers usually reply within 1 day
-
bug needs-triage
Difficulty 3/5 1-2 days Newbie friendliness 68/100
NVIDIA/gpu-operator#2970 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 4/5 3-5 days Newbie friendliness 68/100
NVIDIA/gpu-operator#2957 ·
Maintainers usually reply within 1 day
All issues in NVIDIA/gpu-operator
Similar issues
-
security
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
-
cvss-severity:high devguard l3montree-cybersecurity/...ard-k8s-image-inventory pkg:oci/devguard-k8s-ima...ch=amd64&tag=main-amd64 pkg:oci/devguard-k8s-ima...ch=arm64&tag=main-arm64 risk:low state:open
Difficulty 1/5 Under an hour Newbie friendliness 76/100
l3montree-dev/devguard#3094 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 84/100
JuliusBrussee/caveman#1127 · 1 comment ·
Maintainers usually reply within 1 day
-
enhancement low priority
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
eugenioenko/ttt#674 ·
Maintainers usually reply within 1 day
-
kind/bug
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
gpustack/gpustack-operator#640 ·
Maintainers usually reply within 1 day