ggml-org/llama.cpp
feature request - disabling tokenizer in conversion / inference
Fermée
#1 765 ouverte le 8 juin 2023
enhancementgood first issuehelp wanted
Métriques du dépôt
- Stars
- (124 134 étoiles)
- Métriques de merge PR
- (Merge moyen 6j 8h) (389 PRs mergées en 30 j)
Description
in #1764 i asked if it'd be possible to add a Huggingface tokenizer. but - HF tokenizers are quite flexible and officially supporting them in llama.cpp (or ggml?) might be a lot of hassle.
a much easier workaround would be allowing to disable tokenizers in both model conversion and inference. this means the users are supposed to encode(text)/decode(ids) in their implementation for using llama.cpp. in my case, for example, i'll use a python GUI and a wrapper anyway.
i'd like to work on it, but honestly i don't think i understand enough to be able to do this. i'd appreciate very much if anyone's interested in it.