Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

[Lane 2 · kotlin-core] tokenizer.json parity — apply the normalizer block (Replace, Prepend, Sequence, NFC) before pre-tokenization

Aperta
  • Qualcosa non va?
#1,325 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
1-2 giorni
Idoneità per principianti
43/100
Tipo di issue
Funzionalità
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
kotlin
Ambito
backend, data

Direzione di ricerca

Start with SpecialTokenSplitter.kt and the normalizer handling in SentencePieceTokenizer.kt#L382-L391; the issue names both tokenizer factories to update and the existing source sets for platform NFC code. First check the Lane 1 survey for the NFC strategy, then implement the normalizer plumbing while it is open. Run the listed jvmTest and linuxX64Test; done means the three specified normalizers pass, unknown types fail with their names, and the ModernBERT fixture ends with id 3974.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

coding size:m skill:kotlin-core sub-issue

Sub-issue of #1321 (HF tokenizer.json parity for BPE encoders).

Lane: 2 · Kotlin core
Skill needed: Kotlin Multiplatform (expect/actual or a pure-Kotlin table), Unicode normalization basics. No tensor knowledge.
Size: m (1–2 days; most of it is the NFC implementation and its tests)
Blocked by: the Lane 1 survey (it decides the NFC strategy). Start with the Replace/Sequence plumbing while it is open.

What to do

  1. Add a small Normalizer step to sk.ainet.io.tokenizer that is built from tokenizer.json#normalizer and applied to each non-special segment before pre-tokenization (the reference applies it before added-token matching only for tokens with normalized: true; applying it to the segments between atomic matches is the observable behaviour for the files in the parent).
  2. Support, in this order: null (no-op), Replace (pattern.String → content; pattern.Regex can throw UnsupportedTokenizerException with the pattern in the message), Prepend, Sequence (compose), NFC. Everything else: throw UnsupportedTokenizerException naming the type — do not silently skip, that is the current bug.
  3. Implement NFC the way Lane 1 recommended. If that is expect/actual: JVM via java.text.Normalizer, JS/Wasm via normalize("NFC"), Kotlin/Native passthrough with a documented warning in the KDoc. Put the per-target code in the existing jvmMain / jsMain / wasmJsMain / native64Main source sets of skainet-io-core.
  4. Wire it into QwenByteLevelBpeTokenizer.fromTokenizerJson and SentencePieceTokenizer.fromTokenizerJson; the existing Prepend sniffing at SentencePieceTokenizer.kt#L382-L391 should become a consumer of the new step rather than a second parser.
  5. Tests in commonTest: Replace, Sequence, and NFC on "e\u0301" → "é" (JVM at least; mark the native expectation explicitly). Run ./gradlew :skainet-io:skainet-io-core:jvmTest :skainet-io:skainet-io-core:linuxX64Test.

Acceptance

  • ModernBERT file: the third fixture text of #1321 (ending in decomposed e + U+0301) encodes identically to the reference — last id 3974, not 299, 136, 212
  • Unknown normalizer types fail loudly with the type name in the message
  • Tests green on jvmTest and linuxX64Test
  • Result reported back on #1321

Notes

Sibling file to copy the shape from: SpecialTokenSplitter.kt is a small decorator built by the factory from a JSON block — the normalizer is the same pattern one step earlier in the pipeline. Keep the implementation out of QwenByteLevelBpeTokenizer itself so the Metaspace lane can reuse it.

Lingua principale
Kotlin
Stelle
52
Fork
15
Merge medio
1g 8h
PR unite (30g)
37

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di SKaiNET-developers/SKaiNET

Tutte le issue di SKaiNET-developers/SKaiNET

Issue simili

Altre issue su Kotlin

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.