diag warning after upgrade to 4.2.3
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 25/100
- Issue type
- Bug
- Clarity
- Needs clarification
- Activity status
- Stale
- Domain
- testing-qa
Research direction
The issue names no source file or test. Start by reproducing the DCGM 4.2.3 diagnostic on an affected node, then inspect the DCGM and system logs alongside the PCIe integration output; the work is done when the cause is identified and the diagnostic no longer reports these warnings.
Written by the indexing model from the issue text.
Description
After upgrading to 4.2.3 I now see a lot of warnings on nodes after reboots.
Additional reboots to reset the GPU do not clear these warning:
Executing in Pod nvidia-dcgm-rz9cr on Host 10.0.7.139
Successfully ran diagnostic for group.
+---------------------------+------------------------------------------------+
| Diagnostic | Result |
+===========================+================================================+
|----- Metadata ----------+------------------------------------------------|
| DCGM Version | 4.2.3 |
| Driver Version Detected | 570.148.08 |
| GPU Device IDs Detected | 26b9, 26b9, 26b9, 26b9 |
|----- Deployment --------+------------------------------------------------|
| software | Pass |
| | GPU0: Pass |
| | GPU1: Pass |
| | GPU2: Pass |
| | GPU3: Pass |
+----- Hardware ----------+------------------------------------------------+
| memory | Pass |
| | GPU0: Pass |
| | GPU1: Pass |
| | GPU2: Pass |
| | GPU3: Pass |
+----- Integration -------+------------------------------------------------+
| pcie | Fail |
| Warning | There was an internal error during the test: |
| | 'A child process (152) exited with non-zero s |
| | tatus 1' Check DCGM and system logs for error |
| | s. Reset GPU. Restart DCGM. Rerun diagnostics |
| | . |
| Warning | There was an internal error during the test: |
| | 'A child process (153) exited with non-zero s |
| | tatus 1' Check DCGM and system logs for error |
| | s. Reset GPU. Restart DCGM. Rerun diagnostics |
| | . |
| Warning | There was an internal error during the test: |
| | 'Output of child process (152) couldn't be pa |
| | rsed: '* Line 1, Column 1 Syntax error: val |
| | ue, object or array expected. '' Check DCGM a |
| | nd system logs for errors. Reset GPU. Restart |
| | DCGM. Rerun diagnostics. |
| Warning | There was an internal error during the test: |
| | 'Output of child process (153) couldn't be pa |
| | rsed: '* Line 1, Column 1 Syntax error: val |
| | ue, object or array expected. '' Check DCGM a |
| | nd system logs for errors. Reset GPU. Restart |
| | DCGM. Rerun diagnostics. |
| | GPU0: Pass |
| | GPU1: Pass |
| | GPU2: Pass |
| | GPU3: Pass |
- Dominant language
- C++
- Stars
- 813
- Forks
- 115
- Avg merge
- 6d 10h
- Merged PRs (30d)
- 1
Getting set up
- No Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from NVIDIA/DCGM
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
-
dcgmi InstallCtrlHandler unhandled returnMay be free again A pull request for this issue was closed without being merged. Open
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
Similar issues
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
metasequoiaime/msime#10989 ·
Maintainers usually reply within 1 day
-
documentation good first issue help wanted
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
vek-language/vek-lang#1 · 1 comment ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
eclipse-score/lifecycle#957 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 62/100
mw00/project-maya#877 ·
Maintainers usually reply within 1 day