Hacktoberfest 2026：メンテナが10月に向けて印を付けた、オープンで初心者向けの issue。 Hacktoberfest の issue を見る

[Lane 2 · kotlin-core] tokenizer.json parity — apply the normalizer block (Replace, Prepend, Sequence, NFC) before pre-tokenization

オープン
  • 何か問題がありますか？
#1,325 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

まだ誰も着手していません。

評価

難易度
4/5
見積もり時間
1〜2日
初心者へのやさしさ
43/100
issue の種類
機能追加
明瞭さ
明確に書かれている
活発さ
活発
技術スタック
kotlin
領域
backend, data

調査の方向性

Start with SpecialTokenSplitter.kt and the normalizer handling in SentencePieceTokenizer.kt#L382-L391; the issue names both tokenizer factories to update and the existing source sets for platform NFC code. First check the Lane 1 survey for the NFC strategy, then implement the normalizer plumbing while it is open. Run the listed jvmTest and linuxX64Test; done means the three specified normalizers pass, unknown types fail with their names, and the ModernBERT fixture ends with id 3974.

索引モデルが issue の本文から書いたものです。

説明

coding size:m skill:kotlin-core sub-issue

Sub-issue of #1321 (HF tokenizer.json parity for BPE encoders).

Lane: 2 · Kotlin core
Skill needed: Kotlin Multiplatform (expect/actual or a pure-Kotlin table), Unicode normalization basics. No tensor knowledge.
Size: m (1–2 days; most of it is the NFC implementation and its tests)
Blocked by: the Lane 1 survey (it decides the NFC strategy). Start with the Replace/Sequence plumbing while it is open.

What to do

  1. Add a small Normalizer step to sk.ainet.io.tokenizer that is built from tokenizer.json#normalizer and applied to each non-special segment before pre-tokenization (the reference applies it before added-token matching only for tokens with normalized: true; applying it to the segments between atomic matches is the observable behaviour for the files in the parent).
  2. Support, in this order: null (no-op), Replace (pattern.String → content; pattern.Regex can throw UnsupportedTokenizerException with the pattern in the message), Prepend, Sequence (compose), NFC. Everything else: throw UnsupportedTokenizerException naming the type — do not silently skip, that is the current bug.
  3. Implement NFC the way Lane 1 recommended. If that is expect/actual: JVM via java.text.Normalizer, JS/Wasm via normalize("NFC"), Kotlin/Native passthrough with a documented warning in the KDoc. Put the per-target code in the existing jvmMain / jsMain / wasmJsMain / native64Main source sets of skainet-io-core.
  4. Wire it into QwenByteLevelBpeTokenizer.fromTokenizerJson and SentencePieceTokenizer.fromTokenizerJson; the existing Prepend sniffing at SentencePieceTokenizer.kt#L382-L391 should become a consumer of the new step rather than a second parser.
  5. Tests in commonTest: Replace, Sequence, and NFC on "e\u0301" → "é" (JVM at least; mark the native expectation explicitly). Run ./gradlew :skainet-io:skainet-io-core:jvmTest :skainet-io:skainet-io-core:linuxX64Test.

Acceptance

  • ModernBERT file: the third fixture text of #1321 (ending in decomposed e + U+0301) encodes identically to the reference — last id 3974, not 299, 136, 212
  • Unknown normalizer types fail loudly with the type name in the message
  • Tests green on jvmTest and linuxX64Test
  • Result reported back on #1321

Notes

Sibling file to copy the shape from: SpecialTokenSplitter.kt is a small decorator built by the factory from a JSON block — the normalizer is the same pattern one step earlier in the pipeline. Keep the implementation out of QwenByteLevelBpeTokenizer itself so the Metaspace lane can reuse it.

主要言語
Kotlin
スター
52
フォーク
15
平均マージ
1日 8時間
マージ済み PR（30日）
37

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

SKaiNET-developers/SKaiNET のほかの issue

SKaiNET-developers/SKaiNET の issue をすべて見る

似ている issue

Kotlin の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。