Inference not running when using a tokenizer with word model

Hi!

First and foremost... llama.cpp is absolutely awesome! Great work!

Today, I am facing a problem when using a custom sentencepiece tokenizer. It is a little different than the default Llama one. Instead of using bpe, I am using the word model type. I did this because my vocabulary is < 1000 words, thus I do not need subword splitting.

Training a model with this tokenizer worked like a charm. Inference using hugging face also worked. Quantization via llama.cpp worked as well. Inference using a quantized model, however, did not. Here is the error message in short:

terminate called after throwing an instance of 'std::out_of_range'
  what():  _Map_base::at

To reproduce this issue, I have uploaded a demo model here: https://huggingface.co/TristanBehrens/wordtokenizer

If you just run it with llama.cpp's main like this, you will see the error:

./main -m model.q2_k.gguf -n 128

I have also created a colab notebook, which reproduces the error: https://colab.research.google.com/drive/1NtBTdMx887oROSgAjd_7Eds3MN9itEKm?usp=sharing

I currently feel there is a small issue in how llama.cpp uses the tokenizer. Should be a quick fix. 🤗