Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

[Lane 6 · review] tokenizer.json parity — DARC review

Cerrado
  • ¿Algo no está bien?
#1,330 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
1-2 días
Aptitud para principiantes
18/100
Tipo de issue
Nueva funcionalidad
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
kotlin, python
Área
machine-learning, testing-qa

Línea de trabajo

Read parent issue #1321 and compare each acceptance criterion with the shipped tokenizer code and docs. Regenerate goldens with the pinned tokenizers version using the Lane 4 script, then run the listed Gradle fixture and test tasks and hand-check three cases in Python. Verify the cited semantics and the specified failure behavior, added-token vocabSize, decode fixture, and changelog; report pass or fail on #1321 and update or close #912 as appropriate.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

assessment size:s skill:review sub-issue

Sub-issue of #1321 (HF tokenizer.json parity for BPE encoders).

Lane: 6 · Review
Skill needed: DARC review — must not have implemented any Lane 2 or Lane 4 task. Kotlin reading skills and access to a Python environment to re-run the goldens.
Size: s (a few hours)
Blocked by: all Lane 2 and Lane 4 sub-issues merged.

What to do

  1. D — Document. Re-read the parent's problem statement against the shipped code: are the four gaps closed, or did any lane narrow its scope? Record deviations as comments on the parent.
  2. A — Assess. Regenerate the goldens with the pinned tokenizers version (Lane 4 script) and run ./gradlew :skainet-io:skainet-io-core:downloadTokenizerJsonGoldenFixtures :skainet-io:skainet-io-core:jvmTest :skainet-io:skainet-io-core:linuxX64Test. Spot-check three cases by hand with the Python library.
  3. R — Research. Click every link in the parent, the docs page and the PR descriptions; confirm the tokenizers semantics cited (added-token matching, merges form) still hold at the pinned version.
  4. C — Code. Check that unsupported normalizer / pre-tokenizer types fail loudly, that vocabSize includes added tokens, that the #912 decode fixture is present, and that CHANGELOG.md lists the behaviour changes.
  5. Close #912 if its four asks are met, or comment on it with what remains.

Acceptance

  • Written review comment on #1321 with a pass/fail per acceptance criterion of the parent
  • Docs support matrix matches the code
  • #912 closed or updated
  • Result reported back on #1321 (then the parent can be closed)

Notes

@DarcValidated targets TensorOps functions and does not apply to the tokenizer package; the review comment on the parent is the durable record here.

Lenguaje dominante
Kotlin
Estrellas
52
Forks
15
Merge medio
1 d 1 h
PR fusionados (30 d)
41

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de SKaiNET-developers/SKaiNET

Todos los issues de SKaiNET-developers/SKaiNET

Issues similares

Más issues de Kotlin

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.