Cannot tokenize byte sequences that are not valid UTF-8 due to design flaw

Open
#51 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Bug
Clarity
Needs clarification
Activity status
Stale
Tech stack
rust
Domain
tooling

Research direction

Start by inspecting the bpe-openai encode method and the input type it accepts. Determine how the library should expose arbitrary byte sequences, then add coverage for invalid UTF-8 inputs and verify that all byte sequences can be tokenized.

Written by the indexing model from the issue text.

Description

Hello,

The BPE algorithm is capable of tokenizing any byte sequence, and LLMs generally accept any sequence of tokens and use token dictionaries that can successfully represent any byte sequence, but the encode method in bpe-openai accepts a type that has to be valid UTF-8. So there are lots of byte sequences, many of which are only 1 byte long, which you cannot tokenize using this library.

Dominant language
Rust
Stars
134
Forks
24
Avg merge
16h 27m
Merged PRs (30d)
11

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from github/rust-gems

All issues in github/rust-gems

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.