tryAGI/Tiktoken
High-performance .NET BPE tokenizer — up to 618 MiB/s, competitive with Rust. Zero-allocation counting, multilingual cache, o200k/cl100k/r50k/p50k encodings + HuggingFace tokenizer.json support.
Is tryAGI/Tiktoken beginner-friendly?
We have not seen enough recent pull requests from first-time contributors to tryAGI/Tiktoken to say how often they are merged. 1 beginner-friendly issue is open now.
- Stars
- 91
- Forks
- 7
- Open beginner issues
- 1
- Indexed issues
- 2
- Avg merge
- 14h 51m
- Merged PRs (30d)
- 4
- Dominant language
- C#
- License
- MIT
- Last GitHub push
- Sep 25, 2026
- Latest indexed
- Sep 19, 2026
- Contributing guide
- Contributing guide
- Code of conduct
- Code of conduct
- Beginner labels
- No beginner labels indexed
How to contribute to tryAGI/Tiktoken
- Read the contributing guide first: it says how the maintainers want changes proposed, tested and reviewed.
- Read its code of conduct: it covers issues and pull requests as well as chat.
- Your contributions will be published under the project's MIT license.
- Pick the open beginner-friendly issue below and comment that you want to work on it before you start.
Loading issues
Issues somebody may already be working on are listed last. List everything by date
-
EncodeNative reads post-special-token segments from offset 0 on net8+ (silent token corruption)Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
Difficulty 5/5 Over a week Newbie friendliness 25/100