ggml-org/llama.cpp
GitHub で見るSupport for Sparse MoE models like Camelidae and Sparsetral
Open
#5,365 opened on 2024年2月6日
enhancementgood first issue
Repository metrics
- Stars
- (110,169 stars)
- PR merge metrics
- (平均マージ 6d 8h) (30d で 389 merged PRs)
説明
A new technique for broadening the capabilities of LLMs without massively increasing resource requirements has recently emerged. Instead of shuffling tokens in between expert models like a standard MoE, it shuffles them between expert LoRAs. This gives much of the benefit of a MoE, but with very low memory and compute requirements above what is needed by the base model. Please consider adding support for inferencing and quantizing these sparse MoEs, as they are a very promising branch of LLMs that can be not only inferenced but fully trained on attainable consumer hardware.
Camelidae on HF: https://huggingface.co/hywu
Sparsetral on HF:https://huggingface.co/serpdotai/sparsetral-16x7B-v2