Support for Sparse MoE models like Camelidae and Sparsetral

A new technique for broadening the capabilities of LLMs without massively increasing resource requirements has recently emerged. Instead of shuffling tokens in between expert models like a standard MoE, it shuffles them between expert LoRAs. This gives much of the benefit of a MoE, but with very low memory and compute requirements above what is needed by the base model. Please consider adding support for inferencing and quantizing these sparse MoEs, as they are a very promising branch of LLMs that can be not only inferenced but fully trained on attainable consumer hardware.

Camelidae on HF: https://huggingface.co/hywu

Sparsetral on HF:https://huggingface.co/serpdotai/sparsetral-16x7B-v2