Inference not running when using a tokenizer with word model
#6 717 ouverte le 17 avr. 2024
Métriques du dépôt
- Stars
- (110 169 stars)
- Métriques de merge PR
- (Merge moyen 6j 8h) (389 PRs mergées en 30 j)
Description
Hi!
First and foremost... llama.cpp is absolutely awesome! Great work!
Today, I am facing a problem when using a custom sentencepiece tokenizer. It is a little different than the default Llama one. Instead of using bpe, I am using the word model type. I did this because my vocabulary is < 1000 words, thus I do not need subword splitting.
Training a model with this tokenizer worked like a charm. Inference using hugging face also worked. Quantization via llama.cpp worked as well. Inference using a quantized model, however, did not. Here is the error message in short:
terminate called after throwing an instance of 'std::out_of_range'
what(): _Map_base::at
To reproduce this issue, I have uploaded a demo model here: https://huggingface.co/TristanBehrens/wordtokenizer
If you just run it with llama.cpp's main like this, you will see the error:
./main -m model.q2_k.gguf -n 128
I have also created a colab notebook, which reproduces the error: https://colab.research.google.com/drive/1NtBTdMx887oROSgAjd_7Eds3MN9itEKm?usp=sharing
I currently feel there is a small issue in how llama.cpp uses the tokenizer. Should be a quick fix. 🤗