Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

[Lane 5 · docs] tokenizer.json parity — reference page: tokenizer.json support matrix

Abierto
  • ¿Algo no está bien?
#1,329 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
3/5
Tiempo estimado
1-2 días
Aptitud para principiantes
78/100
Tipo de issue
Documentación
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
kotlin
Área
ai, documentation

Línea de trabajo

Read TokenizerFactory.kt and the Lane 2 measurements from #1321 to fill the support matrix, then follow an existing reference page for its Antora structure. Add tokenizer-json-support.adoc, register it in nav.adoc, and add the KDoc pointer; check the Lane 4 sub-issues for the Python golden script and gated test. Done when the page is listed under Reference, unsupported or partial rows link to open issues, and the result is reported on #1321.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

documentation good first issue size:s skill:docs sub-issue

Sub-issue of #1321 (HF tokenizer.json parity for BPE encoders).

Lane: 5 · Docs
Skill needed: AsciiDoc and the ability to read a Kotlin when statement. No build or tokenizer knowledge.
Size: s (a few hours)
Blocked by: nothing for the first version; update the table as the Lane 2 PRs merge.

What to do

  1. The engine docs have no tokenizer page. Add docs/modules/ROOT/pages/reference/tokenizer-json-support.adoc and register it in docs/modules/ROOT/nav.adoc under Reference.
  2. Content: a support matrix of what sk.ainet.io.tokenizer.TokenizerFactory accepts, one row per tokenizer.json feature — model.type (BPE / Unigram / WordPiece), model.merges form (string / array), byte_fallback, ignore_merges, each normalizer type, each pre_tokenizer type, added_tokens (special: true / false, the four flags), post_processor (not applied — callers add BOS/EOS themselves, say so), decoder. Columns: Supported (yes / no / partial), Since version, Issue link. Derive the current state from TokenizerFactory.kt and the #1321 measurements; link each "no" to its lane sub-issue.
  3. A second short section "GGUF vs tokenizer.json": the factory dispatches per architecture, not per file format (the KDoc at the top of TokenizerFactory.kt already says this well — reuse it).
  4. A "How to verify parity" paragraph pointing to the Python golden script and the gated test from the Lane 4 sub-issues.
  5. Add a one-line pointer from the TokenizerFactory KDoc to the new page. Build the docs locally if you can (docs/README or the Antora playbook in the repo explains how); otherwise a careful preview in the GitHub editor is acceptable for a first PR.

Acceptance

  • Page renders in the Antora nav under Reference
  • Every "no" / "partial" row links to an open issue
  • fromTokenizerJson KDoc links to the page
  • Result reported back on #1321

Notes

Follow the structure of an existing reference page in docs/modules/ROOT/pages/reference/ for headings and the :description: attribute. The downstream repo (SKaiNET-transformers) has explanation/tokenizer-internals.adoc; do not duplicate it, link to it for the algorithm descriptions.

Lenguaje dominante
Kotlin
Estrellas
52
Forks
15
Merge medio
1 d 15 h
PR fusionados (30 d)
36

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de SKaiNET-developers/SKaiNET

Todos los issues de SKaiNET-developers/SKaiNET

Issues similares

Más issues de Kotlin

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.