Auto Power-Limit Settings with Operator Policy
@henry118 is already working on this.
Since Aug 31, 2026.
Assessment
This issue has not been assessed yet.
Description
Describe the Ask
Many GPU(s) in the wild getting py-torched with high heat abuse and they die fast. GPU Operator shall offer a power limiting/regulating feature to avoid over-heat & burn.
To Reproduce
Run your 600Watts Air/Poorly-Cooled GPUs next to each other for model train/fine-tune and observe accumulated temps surpassing >100C Temps.
Requested feature
GPU Operator could able to monitor operating temperature of each GPU and keep it in allowed Temp Range, by using "nvidia-smi -pl" power limiting ability with autonomously if this feature is enabled.
Environment (please provide the following information):
- GPU Operator Version: 25.3.2
- Container Runtime Version: Driver Version: 570.148.08 CUDA Version: 12.8
- Kubernetes Distro and Version: RH OCP 4.19
Information to attach (optional if deemed irrelevant)
sh-5.1# nvidia-smi
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.148.08 Driver Version: 570.148.08 CUDA Version: 12.8 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA GeForce RTX 4090 D On | 00000000:4F:00.0 Off | Off |
| 94% 97C P0 583W / 600W | 47404MiB / 49140MiB | 98% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
| 1 NVIDIA GeForce RTX 4090 D On | 00000000:F5:00.0 Off | Off |
| 30% 36C P8 18W / 400W | 1MiB / 49140MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| 0 N/A N/A 38730 C python 47394MiB |
+-----------------------------------------------------------------------------------------+
sh-5.1# nvidia-smi -pl 400 -i 0
Power limit for GPU 00000000:4F:00.0 was set to 400.00 W from 600.00 W.
All done.
<--Cool Down Period->
sh-5.1# nvidia-smi
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.148.08 Driver Version: 570.148.08 CUDA Version: 12.8 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA GeForce RTX 4090 D On | 00000000:4F:00.0 Off | Off |
| 66% 62C P0 393W / 400W | 47404MiB / 49140MiB | 98% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
| 1 NVIDIA GeForce RTX 4090 D On | 00000000:F5:00.0 Off | Off |
| 30% 35C P8 18W / 400W | 1MiB / 49140MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| 0 N/A N/A 38730 C python 47394MiB |
+-----------------------------------------------------------------------------------------+
sh-5.1#
- Dominant language
- Go
- Stars
- 2.9k
- Forks
- 552
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 88
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from NVIDIA/gpu-operator
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
NVIDIA/gpu-operator#2955 ·
-
lifecycle/stale question
Difficulty 2/5 1-3 hours Newbie friendliness 64/100
NVIDIA/gpu-operator#2280 · 2 comments · 1 reaction ·
-
Difficulty 4/5 3-5 days Newbie friendliness 68/100
NVIDIA/gpu-operator#2957 ·
-
feature lifecycle/frozen needs-triage
Difficulty 3/5 1-2 days Newbie friendliness 65/100
NVIDIA/gpu-operator#2938 ·
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
NVIDIA/gpu-operator#2923 ·
All issues in NVIDIA/gpu-operator
Similar issues
-
Difficulty 1/5 Under an hour Newbie friendliness 90/100
-
Bob Shell support Openenhancement
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
santhosh-tekuri/jsonschema#276 ·