Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

[Lane 4 · numerics] tokenizer.json parity — HF golden encode fixtures for every shipped family that loads a tokenizer.json

Abierto
  • ¿Algo no está bien?
#476 1 comentario 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
55/100
Tipo de issue
Nueva funcionalidad
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
python
Área
machine-learning, testing

Línea de trabajo

Start with the fromTokenizerJson call sites listed in #471 and audit llm-inference/* and llm-runtime/* to identify tokenizer families. Read the five reference fixture rows in #471, then create scripts/tokenizer_goldens.py using pinned tokenizers and huggingface_hub versions. Generate the family and Laya JSON files under tests/tokenizer-goldens/; done means each has the required metadata and cases, and the audit and results are reported on #471.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

coding good first issue size:s skill:numerics sub-issue

Sub-issue of #471 (tokenizer.json parity gate).

Lane: 4 · Ground truth / CI
Skill needed: Python (tokenizers, huggingface_hub), reading Kotlin file names. No Kotlin.
Size: s (a few hours, most of it finding which families ship a tokenizer.json)
Blocked by: nothing

What to do

  1. Audit which model families in llm-inference/* and llm-runtime/* load a Hugging Face tokenizer.json (SafeTensors paths) — start from the call sites listed in #471 and grep for fromTokenizerJson. Expected: Gemma 4 (SafeTensors), Llama 3.2 and Qwen via the kllama CLI, SmolLM2, BitNet, Apertus, Gemma3n, Moonshine, BERT/MiniLM; confirm rather than assume. Post the list as a comment on #471.
  2. Write scripts/tokenizer_goldens.py (or reuse the engine's script of the same name once SKaiNET#1321 lands it): input = Hub repo id, revision, path; output = JSON with repo, revision, path, sha256, tokenizers_version, vocab_size_with_added, and cases of {text, ids, tokens} from Tokenizer.from_file(p).encode(text, add_special_tokens=False).
  3. Texts: the five #471 fixture texts plus ~15 targeting one feature each — a lone special token (<bos>, <|im_start|>), tabs/newlines, a 24-space run, CJK, Arabic, an emoji with skin-tone modifier, decomposed vs composed accents, digits, the empty string, a single space, a chat-template-rendered single turn for chat families.
  4. Generate one golden per audited family at a pinned Hub revision, plus the two Laya files (convaiinnovations/laya @ 7b928d828b7b0e022f929d9bd2e44165aa270148, paths tokenizer/tokenizer.json and multilingual/tokenizer/tokenizer.json). Commit them under tests/tokenizer-goldens/<family>.json. Do not commit the tokenizer files; gated models (Gemma, Llama) are fine because only ids are stored.
  5. Script header: exact pip pins and the command line that regenerates everything.

Acceptance

  • Family audit comment on #471
  • One golden JSON per family that loads a tokenizer.json, plus the two Laya files, each with sha256 and tokenizers_version
  • The five #471 reference rows reproduced byte-for-byte
  • Result reported back on #471

Notes

For gated repositories note HF_TOKEN is required to regenerate; the committed ids are not gated. If a family's tokenizer.json cannot be loaded by tokenizers itself (it happens with hand-edited files), record that in a "note" field — it is a finding for the maintainers.

Lenguaje dominante
Kotlin
Estrellas
8
Forks
3
Merge medio
1 d 16 h
PR fusionados (30 d)
48

Preparar el entorno

Este proyecto no incluye contenedor de desarrollo, Dockerfile ni guía de contribución, así que la configuración corre por tu cuenta: empieza por su README y consulta nuestra guía para la primera contribución para los pasos generales.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de SKaiNET-developers/SKaiNET-transformers

Todos los issues de SKaiNET-developers/SKaiNET-transformers

Issues similares

Más issues de Kotlin

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.