CUDA issue on the Compute Instance
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 25/100
- Issue type
- Bug
- Clarity
- Needs clarification
- Activity status
- Stale
- Tech stack
- python, pytorch
- Domain
- cloud, infrastructure, machine-learning
Research direction
Start by reproducing python -c "import torch; print(torch.cuda.is_available())" on the Compute Instance and comparing it with the working Computer Cluster VMs. Check the reported nvidia-smi, nvcc --version, and CUDA initialization error, then verify that PyTorch detects the A100 GPUs and that the NCCL test compiles and runs.
Written by the indexing model from the issue text.
Description
PyTorch cannot detect GPUs on the Compute Instance with NVIDIA A100 GPUs. nvidia-smi and nvcc --version can return the installed CUDA version and CUDA toolkit versions. But PyTorch returns the following:
# python -c "import torch; print(torch.cuda.is_available())"
/opt/conda/lib/python3.8/site-packages/torch/cuda/__init__.py:80: UserWarning: CUDA initialization: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 802: system not yet initialized (Triggered internally at /opt/pytorch/pytorch/c10/cuda/CUDAFunctions.cpp:112.)
return torch._C._cuda_getDeviceCount() > 0
False
I tried to pull PyTorch container from Nvidia NGC, AML base images, or compile PyTorch using pip/conda with GPU support. All have the same CUDA issue. I also cannot compile NCCL test on the Compute Instance with GPUs.
However, everything works fine on the Computer Cluster VMs. NCCL compiles/runs file and PyTorch has no issue with CUDA. I suspect CUDA is installed but not configured correctly on the Compute Instance VMs.
- Dominant language
- Jupyter Notebook
- Stars
- 4.4k
- Forks
- 2.6k
- PR merge metrics
- No merged PRs in 30d
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from Azure/MachineLearningNotebooks
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Azure/MachineLearningNotebooks#1975 · 1 comment ·
-
duplicates Open
Difficulty 1/5 Under an hour Newbie friendliness 68/100
Azure/MachineLearningNotebooks#1960 ·
-
machine Open
Difficulty 5/5 Over a week Newbie friendliness 10/100
Azure/MachineLearningNotebooks#1987 ·
-
Difficulty 5/5 Over a week Newbie friendliness 25/100
Azure/MachineLearningNotebooks#1985 · 1 comment ·
-
Difficulty 3/5 1-2 days Newbie friendliness 35/100
Azure/MachineLearningNotebooks#1981 · 1 reaction ·
All issues in Azure/MachineLearningNotebooks
Similar issues
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
palladius/rails8-app-on-gcp#141 · 1 comment ·
-
Difficulty 1/5 Under an hour Newbie friendliness 75/100
-
Difficulty 1/5 Under an hour Newbie friendliness 85/100
-
documentation improve or update documentation priority/low triage
Difficulty 2/5 Half a day Newbie friendliness 86/100
warpdotdev/docs#782 · 1 comment ·
-
customer-reported question
Difficulty 1/5 Under an hour Newbie friendliness 80/100
Azure/awesome-azd#1017 ·