`bpe` Python Bindings?
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 25/100
- Issue type
- Feature
- Clarity
- Needs clarification
- Activity status
- Stale
- Domain
- developer-experience
Research direction
The issue names no implementation files or tests. Start by reviewing the bpe implementation and its public API, then determine whether Python bindings already exist and what interface is needed to support the shown encoding and decoding examples. Done would require an agreed binding scope and corresponding Python usage documentation and tests.
Written by the indexing model from the issue text.
Description
Amazing work on bpe!
Wanted to see if there were any known Python bindings a la the snippets from tiktoken below?
import tiktoken
enc = tiktoken.get_encoding("o200k_base")
assert enc.decode(enc.encode("hello world")) == "hello world"
# To get the tokeniser corresponding to a specific model in the OpenAI API:
enc = tiktoken.encoding_for_model("gpt-4o")
- Dominant language
- Rust
- Stars
- 134
- Forks
- 24
- Avg merge
- 16h 27m
- Merged PRs (30d)
- 11
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from github/rust-gems
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
-
Difficulty 5/5 Over a week Newbie friendliness 30/100
-
Difficulty 4/5 3-5 days Newbie friendliness 38/100
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
All issues in github/rust-gems
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
-
issue
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
-
agentic-workflows
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
web-infra-dev/rspack#15847 ·