Hacktoberfest 2026：维护者为十月标记出来的 issue，仍然开放、适合新手。 浏览 Hacktoberfest issue

[Lane 4 · numerics] tokenizer.json parity — HF golden encode fixtures for every shipped family that loads a tokenizer.json

未关闭
  • 哪里不对？
#476 1 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 1 天内回复

还没有人认领这个 Issue。

评估

难度
4/5
预计耗时
3-5 天
新手友好度
55/100
Issue 类型
功能
描述清晰度
描述清楚
活跃度
活跃
技术栈
python
领域
machine-learning, testing

调研方向

Start with the fromTokenizerJson call sites listed in #471 and audit llm-inference/* and llm-runtime/* to identify tokenizer families. Read the five reference fixture rows in #471, then create scripts/tokenizer_goldens.py using pinned tokenizers and huggingface_hub versions. Generate the family and Laya JSON files under tests/tokenizer-goldens/; done means each has the required metadata and cases, and the audit and results are reported on #471.

由索引模型根据 Issue 内容生成。

描述

coding good first issue size:s skill:numerics sub-issue

Sub-issue of #471 (tokenizer.json parity gate).

Lane: 4 · Ground truth / CI
Skill needed: Python (tokenizers, huggingface_hub), reading Kotlin file names. No Kotlin.
Size: s (a few hours, most of it finding which families ship a tokenizer.json)
Blocked by: nothing

What to do

  1. Audit which model families in llm-inference/* and llm-runtime/* load a Hugging Face tokenizer.json (SafeTensors paths) — start from the call sites listed in #471 and grep for fromTokenizerJson. Expected: Gemma 4 (SafeTensors), Llama 3.2 and Qwen via the kllama CLI, SmolLM2, BitNet, Apertus, Gemma3n, Moonshine, BERT/MiniLM; confirm rather than assume. Post the list as a comment on #471.
  2. Write scripts/tokenizer_goldens.py (or reuse the engine's script of the same name once SKaiNET#1321 lands it): input = Hub repo id, revision, path; output = JSON with repo, revision, path, sha256, tokenizers_version, vocab_size_with_added, and cases of {text, ids, tokens} from Tokenizer.from_file(p).encode(text, add_special_tokens=False).
  3. Texts: the five #471 fixture texts plus ~15 targeting one feature each — a lone special token (<bos>, <|im_start|>), tabs/newlines, a 24-space run, CJK, Arabic, an emoji with skin-tone modifier, decomposed vs composed accents, digits, the empty string, a single space, a chat-template-rendered single turn for chat families.
  4. Generate one golden per audited family at a pinned Hub revision, plus the two Laya files (convaiinnovations/laya @ 7b928d828b7b0e022f929d9bd2e44165aa270148, paths tokenizer/tokenizer.json and multilingual/tokenizer/tokenizer.json). Commit them under tests/tokenizer-goldens/<family>.json. Do not commit the tokenizer files; gated models (Gemma, Llama) are fine because only ids are stored.
  5. Script header: exact pip pins and the command line that regenerates everything.

Acceptance

  • Family audit comment on #471
  • One golden JSON per family that loads a tokenizer.json, plus the two Laya files, each with sha256 and tokenizers_version
  • The five #471 reference rows reproduced byte-for-byte
  • Result reported back on #471

Notes

For gated repositories note HF_TOKEN is required to regenerate; the committed ids are not gated. If a family's tokenizer.json cannot be loaded by tokenizers itself (it happens with hand-edited files), record that in a "note" field — it is a finding for the maintainers.

主要语言
Kotlin
星标
8
派生
3
平均合并
1 天 16 小时
30 天内合并 PR
48

环境准备

这个项目没有提供开发容器、Dockerfile 或贡献指南，环境需要你自己搭建：先看它的 README，通用步骤见我们的新手贡献指南。

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

SKaiNET-developers/SKaiNET-transformers 的其他 Issue

查看 SKaiNET-developers/SKaiNET-transformers 的全部 Issue

相似的 Issue

更多 Kotlin Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。