Hacktoberfest 2026：维护者为十月标记出来的 issue，仍然开放、适合新手。 浏览 Hacktoberfest issue

[Lane 2 · kotlin-core] tokenizer.json parity — Metaspace BPE with byte fallback (encode side of #912)

未关闭
  • 哪里不对？
#1,326 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 1 天内回复

还没有人认领这个 Issue。

评估

难度
4/5
预计耗时
1-2 天
新手友好度
55/100
Issue 类型
功能
描述清晰度
描述清楚
活跃度
活跃
技术栈
kotlin
领域
machine-learning

调研方向

Start by reading TokenizerFactory.fromTokenizerJson, QwenByteLevelBpeTokenizer, and the existing bpeMerge implementation; the issue describes the needed Metaspace behavior and dispatch rules. Reuse the byte-token lookup and decode logic in SentencePieceTokenizer, then add the synthetic dispatch test and Moonshine decode fixture in the named tests. Done means the specified fixtures pass on JVM and linuxX64, Qwen tests remain unchanged, and results are reported on #1321.

由索引模型根据 Issue 内容生成。

描述

coding size:m skill:kotlin-core sub-issue

Sub-issue of #1321 (HF tokenizer.json parity for BPE encoders).

Lane: 2 · Kotlin core
Skill needed: Kotlin, and you should understand BPE merging (the existing bpeMerge in QwenByteLevelBpeTokenizer is 20 lines and is the algorithm). Familiarity with the SentencePiece ▁ convention helps.
Size: m (1–2 days)
Blocked by: the array-merges lane (the mmBERT file does not load without it). Independent of the normalizer lane, but the mmBERT file also declares Replace(" " → "▁"), so run the fixture after both land.

What to do

  1. Today TokenizerFactory.fromTokenizerJson sends every model.type: "BPE" file to QwenByteLevelBpeTokenizer, which (a) splits with the GPT-2 regex, (b) maps UTF-8 bytes to the GPT-2 unicode alphabet, (c) merges by rank. For pre_tokenizer: Metaspace files (Llama, Gemma HF exports, mmBERT) steps (a) and (b) are wrong: the alphabet is plain code points with ▁ for space, and characters missing from the vocab fall back to <0xNN> byte tokens when model.byte_fallback is true. Every word boundary in the mmBERT file currently becomes id 241753.
  2. Separate the merge-rank BPE core from the alphabet step. Suggested shape: keep QwenByteLevelBpeTokenizer as the ByteLevel configuration and add a MetaspaceBpeTokenizer (or a strategy parameter) that: replaces " " with ▁; honours prepend_scheme (always → prepend ▁ to the text; first → only to the first segment; never); splits on ▁ boundaries when split: true; runs the same rank-based bpeMerge; and emits <0xNN> ids for any symbol not in the vocab when byte_fallback is true (fuse_unk otherwise). SentencePieceTokenizer already has the <0xNN> lookup and decode — reuse, don't copy.
  3. Dispatch in TokenizerFactory.fromTokenizerJson on pre_tokenizer.type (Metaspace → new class, ByteLevel or absent → existing class, Sequence → inspect members).
  4. Decode: ▁ → space, <0xNN> runs → bytes, matching the file's decoder: Sequence[Replace, ByteFallback, Fuse]. This is the crash reported in #912 (decode: token '▁Ever' contains non-byte-level char U+2581); add that issue's fixture (decode([18274, 1898, 29973, 2]) → "Ever tried?" on the Moonshine v2 file) as a test.
  5. Tests: synthetic Metaspace file in TokenizerFactoryDispatchTest (3-token vocab, one merge, one <0x..> fallback); then the mmBERT fixture via the Lane 4 goldens. Run ./gradlew :skainet-io:skainet-io-core:jvmTest :skainet-io:skainet-io-core:linuxX64Test.

Acceptance

  • mmBERT file (multilingual/tokenizer/tokenizer.json) encodes all five #1321 fixture texts identically to the reference; e.g. "Which department handles this request?" → [12236, 9888, 26446, 736, 3853, 235336]
  • #912 decode fixture passes on JVM and linuxX64Test
  • bosTokenId/eosTokenId come from added_tokens (<bos>=2, <eos>=1 in the mmBERT file)
  • Qwen fixture tests (QwenByteLevelBpeTokenizerFixtureTest) unchanged
  • Result reported back on #1321

Notes

If the refactor forces a new public class or changes the Tokenizer interface, stop and say so on #1321 — that trips the SKEEP trigger ("public Kotlin API") and the maintainers decide whether a short SKEEP is needed before the PR. Llama 3 files additionally set ignore_merges: true (vocab lookup of the whole word before merging); note whether you handled it, a follow-up issue is fine.

主要语言
Kotlin
星标
52
派生
15
平均合并
1 天 15 小时
30 天内合并 PR
36

环境准备

  • 没有 Dockerfile 或 Docker Compose 文件
  • 没有 Pull Request 模板
  • 阅读贡献指南

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

SKaiNET-developers/SKaiNET 的其他 Issue

查看 SKaiNET-developers/SKaiNET 的全部 Issue

相似的 Issue

更多 Kotlin Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。