Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

[Lane 4 · kotlin-core] tokenizer.json parity — gated parity test replaying the goldens through both tokenizer.json entry points

Abierto
  • ¿Algo no está bien?
#477 1 comentario 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
1-2 días
Aptitud para principiantes
64/100
Tipo de issue
Nueva funcionalidad
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
kotlin
Área
testing

Línea de trabajo

Read Gemma4TokenizerParityTest.kt for the self-skip pattern and llm-core/build.gradle.kts for the Gradle task entry point. Add TokenizerJsonGoldenParityTest.kt under llm-core/src/jvmTest/.../tokenizer/, then run the fixture download task and targeted jvmTest command from the issue. Done means both tokenizer JSON entry points are checked against every golden, expected failures are enforced, and the test skips with a notice when fixtures are unavailable.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

coding good first issue size:s skill:kotlin-core sub-issue

Sub-issue of #471 (tokenizer.json parity gate).

Lane: 4 · Ground truth / CI
Skill needed: Kotlin and Gradle; you copy an existing self-skipping test. No tokenizer internals.
Size: s (a few hours)
Blocked by: the golden-generation lane (needs the JSON files)

What to do

  1. Copy the self-skip pattern of Gemma4TokenizerParityTest.kt into a new llm-core/src/jvmTest/kotlin/sk/ainet/apps/llm/tokenizer/TokenizerJsonGoldenParityTest.kt.
  2. Fixture location: $TOKENIZER_GOLDEN_FIXTURES_DIR/<family>/tokenizer.json (environment variable; skip with a printed notice when unset). Add a Gradle task downloadTokenizerGoldenFixtures in llm-core/build.gradle.kts that reads each golden's repo/revision/path, downloads the file from the Hub resolve URL (honouring HF_TOKEN for gated repos) into build/test-fixtures/tokenizer-goldens/<family>/, and checks the sha256. Point the test at that directory by default.
  3. For every golden and for both entry points — TokenizerFactory.fromTokenizerJsonString and GGUFTokenizer.fromTokenizerJson — assert either (a) every case's encode(text) == ids, or (b) the entry point throws UnsupportedTokenizerException (allowed only for fromTokenizerJsonString, see the fail-fast lane). Any other exception is a failure. The assertion message prints text, expected ids, expected token strings and actual ids.
  4. Support an optional per-case "expectedToFail": {"<entry point>": "<issue url>"} marker in the golden JSON: such a case must still fail, and the message tells the developer to remove the marker once it passes. This lets the test merge today and become the progress bar for the Lane 2 fixes and the engine lanes.
  5. Run ./gradlew :llm-core:downloadTokenizerGoldenFixtures :llm-core:jvmTest --tests '*TokenizerJsonGoldenParityTest*'; add the task to the smoke-reference.yml workflow's model-gated job if the maintainers agree (ask on the PR).

Acceptance

  • Test self-skips without fixtures; passes with fixtures and the current markers
  • Both entry points are exercised for every golden
  • Marker mechanism documented in the test's KDoc
  • Result reported back on #471

Notes

Gemma4TokenizerParityTest stays as is (it also checks atomic specials with hand-written ids); the new test is the general gate. Keep it in jvmTest: it needs the filesystem and network-fetched fixtures.

Lenguaje dominante
Kotlin
Estrellas
8
Forks
3
Merge medio
1 d 16 h
PR fusionados (30 d)
48

Preparar el entorno

Este proyecto no incluye contenedor de desarrollo, Dockerfile ni guía de contribución, así que la configuración corre por tu cuenta: empieza por su README y consulta nuestra guía para la primera contribución para los pasos generales.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de SKaiNET-developers/SKaiNET-transformers

Todos los issues de SKaiNET-developers/SKaiNET-transformers

Issues similares

Más issues de Kotlin

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.