[Lane 2 · kotlin-core] tokenizer.json parity — apply the normalizer block (Replace, Prepend, Sequence, NFC) before pre-tokenization
维护者通常 1 天内回复
还没有人认领这个 Issue。
评估
调研方向
Start with SpecialTokenSplitter.kt and the normalizer handling in SentencePieceTokenizer.kt#L382-L391; the issue names both tokenizer factories to update and the existing source sets for platform NFC code. First check the Lane 1 survey for the NFC strategy, then implement the normalizer plumbing while it is open. Run the listed jvmTest and linuxX64Test; done means the three specified normalizers pass, unknown types fail with their names, and the ModernBERT fixture ends with id 3974.
由索引模型根据 Issue 内容生成。
描述
Sub-issue of #1321 (HF tokenizer.json parity for BPE encoders).
Lane: 2 · Kotlin core
Skill needed: Kotlin Multiplatform (expect/actual or a pure-Kotlin table), Unicode normalization basics. No tensor knowledge.
Size: m (1–2 days; most of it is the NFC implementation and its tests)
Blocked by: the Lane 1 survey (it decides the NFC strategy). Start with the Replace/Sequence plumbing while it is open.
What to do
- Add a small
Normalizerstep tosk.ainet.io.tokenizerthat is built fromtokenizer.json#normalizerand applied to each non-special segment before pre-tokenization (the reference applies it before added-token matching only for tokens withnormalized: true; applying it to the segments between atomic matches is the observable behaviour for the files in the parent). - Support, in this order:
null(no-op),Replace(pattern.String→content;pattern.Regexcan throwUnsupportedTokenizerExceptionwith the pattern in the message),Prepend,Sequence(compose),NFC. Everything else: throwUnsupportedTokenizerExceptionnaming the type — do not silently skip, that is the current bug. - Implement NFC the way Lane 1 recommended. If that is
expect/actual: JVM viajava.text.Normalizer, JS/Wasm vianormalize("NFC"), Kotlin/Native passthrough with a documented warning in the KDoc. Put the per-target code in the existingjvmMain/jsMain/wasmJsMain/native64Mainsource sets ofskainet-io-core. - Wire it into
QwenByteLevelBpeTokenizer.fromTokenizerJsonandSentencePieceTokenizer.fromTokenizerJson; the existingPrependsniffing at SentencePieceTokenizer.kt#L382-L391 should become a consumer of the new step rather than a second parser. - Tests in
commonTest:Replace,Sequence, and NFC on"e\u0301"→"é"(JVM at least; mark the native expectation explicitly). Run./gradlew :skainet-io:skainet-io-core:jvmTest :skainet-io:skainet-io-core:linuxX64Test.
Acceptance
- ModernBERT file: the third fixture text of #1321 (ending in decomposed
e+ U+0301) encodes identically to the reference — last id3974, not299, 136, 212 - Unknown normalizer types fail loudly with the type name in the message
- Tests green on
jvmTestandlinuxX64Test - Result reported back on #1321
Notes
Sibling file to copy the shape from: SpecialTokenSplitter.kt is a small decorator built by the factory from a JSON block — the normalizer is the same pattern one step earlier in the pipeline. Keep the implementation out of QwenByteLevelBpeTokenizer itself so the Metaspace lane can reuse it.
- 主要语言
- Kotlin
- 星标
- 52
- 派生
- 15
- 平均合并
- 1 天 15 小时
- 30 天内合并 PR
- 36
环境准备
- 没有 Dockerfile 或 Docker Compose 文件
- 没有 Pull Request 模板
- 阅读贡献指南
从这里开始
- 先读完整个 Issue，再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库，在一个分支上完成修改。
- 提交 Pull Request，并在描述里引用这个 Issue 编号。
SKaiNET-developers/SKaiNET 的其他 Issue
-
coding good first issue size:xs skill:kotlin-core sub-issue
难度 2/5 1-3 小时 新手友好度 88/100
SKaiNET-developers/SKaiNET#1323 ·
维护者通常 1 天内回复
-
coding good first issue platform size:xs skill:js sub-issue
难度 2/5 1-3 小时 新手友好度 72/100
SKaiNET-developers/SKaiNET#1232 ·
维护者通常 1 天内回复
-
skeep tracking
难度 5/5 一周以上 新手友好度 15/100
SKaiNET-developers/SKaiNET#1331 ·
维护者通常 1 天内回复
-
assessment size:s skill:review sub-issue
难度 4/5 1-2 天 新手友好度 18/100
SKaiNET-developers/SKaiNET#1330 ·
维护者通常 1 天内回复
-
documentation good first issue size:s skill:docs sub-issue
难度 3/5 1-2 天 新手友好度 78/100
SKaiNET-developers/SKaiNET#1329 ·
维护者通常 1 天内回复
查看 SKaiNET-developers/SKaiNET 的全部 Issue
相似的 Issue
-
难度 2/5 1-3 小时 新手友好度 83/100
Kotlin/kotlin-lsp#288 ·
-
Cassiopeia drops the `präsentiert von` credit printed in the event body可能已有人在做 关联的 PR 仍在进行中或已合并。 未关闭afk-ok area:data-quality importer size:S
难度 2/5 1-3 小时 新手友好度 77/100
enorm-labs/event-junkie#3166 ·
维护者通常 1 天内回复
-
enhancement good first issue
难度 2/5 1-3 小时 新手友好度 84/100
rajpreetcodes/Skills_MCP_Server#25 ·
维护者通常 1 天内回复
-
bugbane/qf good first issue
难度 2/5 1-3 小时 新手友好度 64/100
osservatorionessuno/bugbane#173 ·
维护者通常 1 天内回复
-
Android: build fails with Android Gradle Plugin 9 (kotlinOptions, lintOptions, proguard-android.txt)未关闭
难度 2/5 1-3 小时 新手友好度 78/100
capacitor-community/stripe#544 ·
维护者通常 1 天内回复