Feature Description

Include (at minimum) eos_token and bos_token keys the huggingface tokenizer_config.json as gguf metadata keys.

Motivation

For chat models these differ from the normal eos and bos tokens and are required to stop the model generating user message tokens.

For example to correctly stop generation of chatml chat models you need to watch for the <|im_end|> token specified in the tokenizer_config.json below.

https://huggingface.co/mlabonne/NeuralBeagle14-7B/blob/main/tokenizer_config.json

however the gguf metadata doesn't include this token

using the chat_format alone without the eos_token causes generation to continue incorrectly past the end of the assistant respone.

Possible Implementation

I'm not too familiar with the gguf-py package but I think adding these to the SpecialVocab class and updating the _try_load_from_tokenizer_json would be a start. Not sure what the correct gguf key should be for these so as to not create confusion.