Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

[Lane 4 · numerics] tokenizer.json parity — HF golden encode fixtures for every shipped family that loads a tokenizer.json

Aperta
  • Qualcosa non va?
#476 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
55/100
Tipo di issue
Funzionalità
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
python
Ambito
machine-learning, testing

Direzione di ricerca

Start with the fromTokenizerJson call sites listed in #471 and audit llm-inference/* and llm-runtime/* to identify tokenizer families. Read the five reference fixture rows in #471, then create scripts/tokenizer_goldens.py using pinned tokenizers and huggingface_hub versions. Generate the family and Laya JSON files under tests/tokenizer-goldens/; done means each has the required metadata and cases, and the audit and results are reported on #471.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

coding good first issue size:s skill:numerics sub-issue

Sub-issue of #471 (tokenizer.json parity gate).

Lane: 4 · Ground truth / CI
Skill needed: Python (tokenizers, huggingface_hub), reading Kotlin file names. No Kotlin.
Size: s (a few hours, most of it finding which families ship a tokenizer.json)
Blocked by: nothing

What to do

  1. Audit which model families in llm-inference/* and llm-runtime/* load a Hugging Face tokenizer.json (SafeTensors paths) — start from the call sites listed in #471 and grep for fromTokenizerJson. Expected: Gemma 4 (SafeTensors), Llama 3.2 and Qwen via the kllama CLI, SmolLM2, BitNet, Apertus, Gemma3n, Moonshine, BERT/MiniLM; confirm rather than assume. Post the list as a comment on #471.
  2. Write scripts/tokenizer_goldens.py (or reuse the engine's script of the same name once SKaiNET#1321 lands it): input = Hub repo id, revision, path; output = JSON with repo, revision, path, sha256, tokenizers_version, vocab_size_with_added, and cases of {text, ids, tokens} from Tokenizer.from_file(p).encode(text, add_special_tokens=False).
  3. Texts: the five #471 fixture texts plus ~15 targeting one feature each — a lone special token (<bos>, <|im_start|>), tabs/newlines, a 24-space run, CJK, Arabic, an emoji with skin-tone modifier, decomposed vs composed accents, digits, the empty string, a single space, a chat-template-rendered single turn for chat families.
  4. Generate one golden per audited family at a pinned Hub revision, plus the two Laya files (convaiinnovations/laya @ 7b928d828b7b0e022f929d9bd2e44165aa270148, paths tokenizer/tokenizer.json and multilingual/tokenizer/tokenizer.json). Commit them under tests/tokenizer-goldens/<family>.json. Do not commit the tokenizer files; gated models (Gemma, Llama) are fine because only ids are stored.
  5. Script header: exact pip pins and the command line that regenerates everything.

Acceptance

  • Family audit comment on #471
  • One golden JSON per family that loads a tokenizer.json, plus the two Laya files, each with sha256 and tokenizers_version
  • The five #471 reference rows reproduced byte-for-byte
  • Result reported back on #471

Notes

For gated repositories note HF_TOKEN is required to regenerate; the committed ids are not gated. If a family's tokenizer.json cannot be loaded by tokenizers itself (it happens with hand-edited files), record that in a "note" field — it is a finding for the maintainers.

Lingua principale
Kotlin
Stelle
8
Fork
3
Merge medio
12h 43m
PR unite (30g)
39

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di SKaiNET-developers/SKaiNET-transformers

Tutte le issue di SKaiNET-developers/SKaiNET-transformers

Issue simili

Altre issue su Kotlin

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.