ggml_cuda_init: failed to initialize CUDA: (null) on Windows with CUDA 12.9
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 35/100
- Issue type
- Bug
- Clarity
- Mostly clear
- Activity status
- Stale
- Domain
- ai-infra-agents, backend
Research direction
The issue is about CUDA initialization failure on Windows with CUDA 12.9. Start by examining the build logs and the CMake configuration (GGML_CUDA flag). Check the ggml_cuda_init function in the llama.cpp source code to understand the CUDA runtime checks. Verify CUDA toolkit installation, driver compatibility, and environment variables. Running a simple CUDA test program outside the library can help isolate the issue.
Written by the indexing model from the issue text.
Description
System Information:
- OS: Windows
- GPU: NVIDIA GeForce RTX 5060 Ti
- NVIDIA Driver Version: 577.00
- CUDA Version (from
nvidia-smi): 12.9 - Python Version: 3.12
- Visual Studio: Visual Studio 2019 with "Desktop development with C++" workload
Problem Description:
I am unable to get llama-cpp-python to use my GPU. When I run a script to load a model with n_gpu_layers=-1, I get the error ggml_cuda_init: failed to initialize CUDA: (null), and all layers are
loaded on the CPU.
Troubleshooting Steps Taken:
- Installed llama-cpp-python using the following command in the "x64 Native Tools Command Prompt for VS 2019" with a Python virtual environment activated:
1 set CMAKE_ARGS="-DGGML_CUDA=on" && pip install --upgrade --force-reinstall llama-cpp-python --no-cache-dir - Verified that the command completes successfully, but the resulting installation does not use the GPU.
- Tried using the deprecated LLAMA_CUBLAS flag, which resulted in a build error (as expected).
- Performed a full cleanup of the environment:
- pip uninstall llama-cpp-python
- pip cache purge
- Manually deleted leftover ~* directories from site-packages.
- Reinstalled after the cleanup, but the problem persists.
- Installed PyTorch with CUDA 12.1 support (pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121) before reinstalling llama-cpp-python, but this did not
resolve the issue. - Confirmed that the correct Python interpreter and virtual environment are being used.
- The run_with_llama_cpp.py script being used is:
1 from llama_cpp import Llama
2
3 llm = Llama(
4 model_path="models/mistral-7b-instruct-v0.2.Q4_K_M.gguf",
5 n_gpu_layers=-1,
6 n_ctx=4096,
7 verbose=True
8 )
9
10 output = llm(
11 "AI is going to ",
12 max_tokens=32,
13 stop=["."],
14 echo=True
15 )
16
17 print(output)
Request:
Could you please provide any insights into why the CUDA initialization might be failing, or suggest any further diagnostic steps? I can provide the full verbose build log if needed.
- Dominant language
- Python
- Stars
- 10.7k
- Forks
- 1.5k
- Avg merge
- 1h 40m
- Merged PRs (30d)
- 5
Getting set up
- No Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from abetlen/llama-cpp-python
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
abetlen/llama-cpp-python#2424 ·
Maintainers usually reply within 1 day
-
Seven llama_sampler_init_* bindings admit keyword arguments that the ctypes function object silently dropsPossibly taken @Belal0066 claimed this 59 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
abetlen/llama-cpp-python#2371 ·
Maintainers usually reply within 1 day
-
uv add llama-cpp-python wheels fails for versions above 0.3.30Possibly taken A pull request linked to this issue is open or already merged. Open
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
abetlen/llama-cpp-python#2352 · 1 comment · 3 reactions ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
abetlen/llama-cpp-python#2314 ·
Maintainers usually reply within 1 day
-
Llama.embed() calls LlamaBatch.add_sequence with old 3-arg signature; missing logits_arrayMay be free again @lxcxjxhx claimed this 136 days ago, and no pull request is open. Open
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
abetlen/llama-cpp-python#2211 · 2 comments ·
Maintainers usually reply within 1 day
All issues in abetlen/llama-cpp-python
Similar issues
-
Bug IronPython Python 3
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
pyrevitlabs/pyRevit#4184 ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 74/100
jqueryscript/awesome-claude-code#1009 ·
-
tests
Difficulty 2/5 1-3 hours Newbie friendliness 66/100
Maintainers usually reply within 3 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Maintainers usually reply within 1 day
-
[Plugin] 群聊记忆 · 人格养成Open
Difficulty 1/5 1-3 hours Newbie friendliness 74/100