huggingface/tokenizers

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production

RustStars 10304Forks 1006Watchers 10304Open issues 128License Apache License 2.0

Details

仓库信息

Ownerhuggingface

Last pushed2025-12-02

Last updated2025-12-15

Issues fetched at—

Community at a glance