Phi-3 mini 4k instruct with MICROSOFT's quantization
#2.273 aberto em 4 de mai. de 2024
Métricas do repositório
- Stars
- (16.227 stars)
- Métricas de merge de PR
- (Mesclagem média 4d) (2 fundiu PRs em 30d)
Description
⚙️ Request New Models
- Link to an existing implementation (e.g. Hugging Face/Github): https://huggingface.co/microsoft/Phi-3-mini-4k-instruct-gguf
- Is this model architecture supported by MLC-LLM? Yes
Additional context
I know others have made this request already (https://github.com/mlc-ai/mlc-llm/issues/2246, https://github.com/mlc-ai/mlc-llm/pull/2222, https://github.com/mlc-ai/mlc-llm/issues/2238, https://github.com/mlc-ai/mlc-llm/issues/2205).
But I am requesting something different: I am suggesting that you do not quantize or modify the weights of the model but that you instead use Microsoft's already 4-bit quantized weights.
The reason is that I suspect (although it is not explicit in their repo) they used quantization-aware training to build these GGUF files. I have tested the regular 32-bit model vs the GGUF 4-bit one and the performance is almost equivalent which is not what I've seen so far with MLC's quantized models (they tend to be more inaccurate compared to their 32-bit counterparts).
Is there a way to use Microsoft's own quantized weights?
Thank you! Federico