Goal

Tokenize and detokenize text for popular BPE and LLM vocabularies (tiktoken/BERT/GPT/etc). Input: text; output: tokens (and reverse).

Scope/Features

• Text in, tokens out (and reverse)
• Configurable vocabulary (select via param)
• Optional: return byte offsets for each token

Rough WIT shape

• tokenize(input: string, vocab: string) -> list<u32>
• detokenize(tokens: list<u32>, vocab: string) -> string

Candidate crates

• tokenizers
• tiktoken-rs

Labels

• good first issue

---