Hacktoberfest 2026：メンテナが10月に向けて印を付けた、オープンで初心者向けの issue。 Hacktoberfest の issue を見る

[Lane 2 · kotlin-core] tokenizer.json parity — match non-special added_tokens atomically and count them in vocabSize

クローズ
  • 何か問題がありますか？
#1,324 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

まだ誰も着手していません。

評価

難易度
3/5
見積もり時間
1〜3時間
初心者へのやさしさ
72/100
issue の種類
機能追加
明瞭さ
明確に書かれている
活発さ
活発
技術スタック
kotlin
領域
tooling

調査の方向性

Read the added-token loop in QwenByteLevelBpeTokenizer.kt and TokenizerFactory.kt, then inspect the related tests in TokenizerFactoryDispatchTest.kt. Add a synthetic case where a non-special token would otherwise split, and check that it encodes atomically and vocabSize includes its id. Run ./gradlew :skainet-io:skainet-io-core:jvmTest; done means the test passes and the changelog records the vocabSize change.

索引モデルが issue の本文から書いたものです。

説明

coding good first issue size:s skill:kotlin-core sub-issue

Sub-issue of #1321 (HF tokenizer.json parity for BPE encoders).

Lane: 2 · Kotlin core
Skill needed: Kotlin. Helpful to have read how tokenizers matches added tokens (one page of docs, linked below). No tensor knowledge.
Size: s (a few hours)
Blocked by: nothing (independent of the merges lane; both touch the same file, so rebase once)

What to do

  1. Read how the reference matches added tokens: every entry of added_tokens is matched atomically, before pre-tokenization, regardless of special. The special flag only decides whether the token is dropped by decode(skip_special_tokens=True). See the AddedVocabulary section of the tokenizers docs: https://huggingface.co/docs/tokenizers/api/added-tokens
  2. In QwenByteLevelBpeTokenizer.kt#L257-L267 the loop skips entries with "special": false. Register all entries in the atomic-match map. Keep a separate Set<Int> of the ids that are marked special so a later change can implement skip_special_tokens; nothing else needs that set today.
  3. Make the same change in TokenizerFactory.kt#L155-L169 for the Unigram path (wrapSentencePieceWithSpecialsFromJson).
  4. Added tokens may have ids above model.vocab (ModernBERT: [UNK]=50280 … and the whitespace runs up to 50367). Extend the tokens array so vocabSize equals the reference get_vocab_size(with_added_tokens=True) — 50368 for ModernBERT instead of today's 50280 — and so decode can return their text.
  5. Tests, in TokenizerFactoryDispatchTest.kt: a synthetic file with one special: false added token whose content would otherwise BPE-split; assert it encodes to its id and that vocabSize counts it. Run ./gradlew :skainet-io:skainet-io-core:jvmTest.

Acceptance

  • Synthetic test: non-special added token encodes atomically; vocabSize includes ids above model.vocab
  • With the ModernBERT file (tokenizer/tokenizer.json) and merges loading (sibling lane), " leading" encodes to [50275, 16378] (reference: [' ', 'leading']) instead of [245, 4283, …], and vocabSize == 50368
  • CHANGELOG.md notes the vocabSize behaviour change
  • Result reported back on #1321

Notes

The flags single_word, lstrip, rstrip, normalized exist on every entry. In the two Laya files they are all false except normalized: true on the non-special ModernBERT entries (and lstrip: true on one special token). Implement plain longest-match as the existing special-token code does; if a file you care about sets single_word or lstrip/rstrip, open a separate issue rather than growing this one. #912 asks for the same vocabSize fix on the SentencePiece side — closing that part here is fine, say so in the PR.

主要言語
Kotlin
スター
52
フォーク
15
平均マージ
1日 8時間
マージ済み PR（30日）
37

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

SKaiNET-developers/SKaiNET のほかの issue

SKaiNET-developers/SKaiNET の issue をすべて見る

似ている issue

Kotlin の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。