Compile bug: Error building with with CUDA 11.8: cudaGridDependencySynchronize undefined in common.cuh

Start with ggml/src/ggml-cuda/common.cuh around the reported cudaGridDependencySynchronize and cudaTriggerProgrammaticLaunchCompletion references, then inspect the CUDA version checks introduced by commit e94722822. Reproduce with CUDA 11.8 using the issue's CMake command and verify that the build completes without undefined-identifier errors.

Git commit

1c3c9674d, current as of August 4

Operating systems

Linux

GGML backends

CUDA

Problem description & steps to reproduce

I'm building with CUDA 11.8 and g++ 11.4.0 on Ubuntu 22.04.
My CMake flags are -DGGML_CUDA=ON and -DGGML_CUDA_NCCL=ON -DGGML_CCACHE=OFF

The build fails around 29% with:

  /home/michael/ai/llama.cpp/ggml/src/ggml-cuda/common.cuh(126): error: identifier "cudaGridDependencySynchronize" is undefined
  /home/michael/ai/llama.cpp/ggml/src/ggml-cuda/common.cuh(132): error: identifier "cudaTriggerProgrammaticLaunchCompletion" is undefined

It looks like they are only declared in cuda_device_runtime_api.h starting with CUDA 12.6, but GGML_CUDA_USE_PDL is enabled for 11.8.

Thank you to the whole llama.cpp team for all the great work you do!

First Bad Commit

GIT bisect found the breaking commit, e94722822 (PR 22522)

Compile command

rm -rf build
cmake -B build -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON -DGGML_CUDA_NCCL=ON -DGGML_CCACHE=OFF
cmake --build build --parallel

Relevant log output

[ 28%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/template-instances/fattn-vec-instance-q8_0-q8_0.cu.o                                                                                                                                                                                 
[ 28%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/template-instances/fattn-vec-instance-bf16-bf16.cu.o                                                                                                                                                                                 
/home/michael/ai/llama.cpp/ggml/src/ggml-cuda/common.cuh(121): error: identifier "cudaGridDependencySynchronize" is undefined                                                                                                                                                                                
                                                                                                                                                                                                                                                                                                             
/home/michael/ai/llama.cpp/ggml/src/ggml-cuda/common.cuh(127): error: identifier "cudaTriggerProgrammaticLaunchCompletion" is undefined                                                                                                                                                                      
                                                                                                                                                                                                                                                                                                             
/home/michael/ai/llama.cpp/ggml/src/ggml-cuda/common.cuh(121): error: identifier "cudaGridDependencySynchronize" is undefined                                                                                                                                                                                
                                                                                                                                                                                                                                                                                                             
/home/michael/ai/llama.cpp/ggml/src/ggml-cuda/common.cuh(127): error: identifier "cudaTriggerProgrammaticLaunchCompletion" is undefined

... and so on