enhancementgood first issuehelp wantedhigh priorityinactive
Repository metrics
- Stars
- (28,442 stars)
- PR merge metrics
- (平均マージ 2d 1h) (30d で 1,000 merged PRs)
説明
cuDNN provides very fast attention implementation and it is well maintained by NVIDIA. We would like to add a new attention backend based on cudnn.
Steps
- Learn this cudnn paged attention python api. https://github.com/NVIDIA/cudnn-frontend/blob/v1.8.0/samples/python/52_scaled_dot_product_attention_with_paged_caches.ipynb
- Add a new attention backend "cudnn" here https://github.com/sgl-project/sglang/tree/main/python/sglang/srt/layers/attention
- We should be able to use it with
python3 -m sglang.launch_server --model meta-llama/Llama-3.1-8B-Instruct --attention-backend cudnn