Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

[Lane 2 · kotlin-core] tokenizer.json parity — accept the array form of model.merges

Abierto Apto para principiantes
  • ¿Algo no está bien?
#1,323 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
2/5
Tiempo estimado
1-3 horas
Aptitud para principiantes
88/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
kotlin
Área
ai, backend

Línea de trabajo

Abre skainet-io/skainet-io-core/src/commonMain/kotlin/sk/ainet/io/tokenizer/QwenByteLevelBpeTokenizer.kt alrededor de las líneas 243–255 y lee cómo se analizan las entradas de merge. Después, inspecciona skainet-io/skainet-io-core/src/commonTest/kotlin/sk/ainet/io/tokenizer/TokenizerFactoryDispatchTest.kt, especialmente la prueba existente de dispatch BPE de Tokenizer JSON. La tarea estará terminada cuando se carguen los merges en formato de array y se codifique la muestra como [3, 2], los arrays mal formados lancen una IllegalArgumentException clara y pasen las pruebas JVM.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

coding good first issue size:xs skill:kotlin-core sub-issue

Sub-issue of #1321 (HF tokenizer.json parity for BPE encoders).

Lane: 2 · Kotlin core
Skill needed: Kotlin and kotlinx.serialization.json basics. No tokenizer or tensor knowledge needed — the fix is a when branch.
Size: xs (under an hour including the test)
Blocked by: nothing

What to do

  1. Open QwenByteLevelBpeTokenizer.kt#L243-L255. Today each merge entry is read with el.jsonPrimitive.content and split on the first space. the Hugging Face tokenizers Python library (PyPI) writes merges as two-element arrays since its 0.20 release, [["Ġ", "Ġ"], ["a", "b"], …], so jsonPrimitive throws IllegalArgumentException: Element class kotlinx.serialization.json.JsonArray … is not a JsonPrimitive and the whole tokenizer fails to load.
  2. Add an is JsonArray branch: require exactly two elements and map them to first to second. Keep the string branch as is (older files and GGUF still use "a b"). Keep the existing is JsonObject -> error(...) branch.
  3. In TokenizerFactoryDispatchTest.kt copy the test tokenizer_json BPE dispatches to byte level BPE and change only "merges": ["a b"] to "merges": [["a", "b"]]. Expected ids for "ab<|end|>" are the same, [3, 2]. Add a second test asserting that a one-element or three-element array throws a clear IllegalArgumentException.
  4. Run ./gradlew :skainet-io:skainet-io-core:jvmTest (fast) and, if you have the toolchain, :skainet-io:skainet-io-core:linuxX64Test.
  5. Add a line under [Unreleased] in CHANGELOG.md and open a PR from feature/<issue-number>-tokenizer-json-array-merges referencing this issue.

Acceptance

  • The synthetic array-form file in the new test loads and encodes to [3, 2]
  • Both published Laya files (ModernBERT, mmBERT) load without rewriting their merges (ids are still wrong for other reasons — that is the sibling lanes' job, not this one's)
  • Existing jvmTest green
  • Result reported back on #1321

Notes

Reproduction with a 0.57.0 BOM project: feed the JSON below to sk.ainet.io.tokenizer.TokenizerFactory.fromTokenizerJson; it throws today and should return a tokenizer that encodes ab<|end|> to [3, 2].

{"version":"1.0","added_tokens":[{"id":2,"content":"<|end|>","special":true}],
 "model":{"type":"BPE","vocab":{"a":0,"b":1,"<|end|>":2,"ab":3},"merges":[["a","b"]]}}

Why both forms matter: the string form is ambiguous when a token itself contains a space; the array form is what every tokenizers release since 0.20 writes (verified with 0.20.3 and the current 0.23.3), while the string form is still accepted on read. The GGUF path (fromGgufFields) is unaffected — GGUF always stores "a b".

Lenguaje dominante
Kotlin
Estrellas
52
Forks
15
Merge medio
1 d 15 h
PR fusionados (30 d)
36

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de SKaiNET-developers/SKaiNET

Todos los issues de SKaiNET-developers/SKaiNET

Issues similares

Más issues de Kotlin

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.