enhancementgood first issuehelp wantedhigh priorityinactive
倉庫指標
- Star
- (28,442 star)
- PR 合併指標
- (平均合併 2天 1小時) (30 天內合併 1,000 個 PR)
描述
cuDNN provides very fast attention implementation and it is well maintained by NVIDIA. We would like to add a new attention backend based on cudnn.
Steps
- Learn this cudnn paged attention python api. https://github.com/NVIDIA/cudnn-frontend/blob/v1.8.0/samples/python/52_scaled_dot_product_attention_with_paged_caches.ipynb
- Add a new attention backend "cudnn" here https://github.com/sgl-project/sglang/tree/main/python/sglang/srt/layers/attention
- We should be able to use it with
python3 -m sglang.launch_server --model meta-llama/Llama-3.1-8B-Instruct --attention-backend cudnn