Hacktoberfest 2026：维护者为十月标记出来的 issue，仍然开放、适合新手。 浏览 Hacktoberfest issue

[Lane 4 · kotlin-core] tokenizer.json parity — gated parity test replaying the goldens through both tokenizer.json entry points

未关闭
  • 哪里不对？
#477 1 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 1 天内回复

还没有人认领这个 Issue。

评估

难度
4/5
预计耗时
1-2 天
新手友好度
64/100
Issue 类型
功能
描述清晰度
描述清楚
活跃度
活跃
技术栈
kotlin
领域
testing

调研方向

Read Gemma4TokenizerParityTest.kt for the self-skip pattern and llm-core/build.gradle.kts for the Gradle task entry point. Add TokenizerJsonGoldenParityTest.kt under llm-core/src/jvmTest/.../tokenizer/, then run the fixture download task and targeted jvmTest command from the issue. Done means both tokenizer JSON entry points are checked against every golden, expected failures are enforced, and the test skips with a notice when fixtures are unavailable.

由索引模型根据 Issue 内容生成。

描述

coding good first issue size:s skill:kotlin-core sub-issue

Sub-issue of #471 (tokenizer.json parity gate).

Lane: 4 · Ground truth / CI
Skill needed: Kotlin and Gradle; you copy an existing self-skipping test. No tokenizer internals.
Size: s (a few hours)
Blocked by: the golden-generation lane (needs the JSON files)

What to do

  1. Copy the self-skip pattern of Gemma4TokenizerParityTest.kt into a new llm-core/src/jvmTest/kotlin/sk/ainet/apps/llm/tokenizer/TokenizerJsonGoldenParityTest.kt.
  2. Fixture location: $TOKENIZER_GOLDEN_FIXTURES_DIR/<family>/tokenizer.json (environment variable; skip with a printed notice when unset). Add a Gradle task downloadTokenizerGoldenFixtures in llm-core/build.gradle.kts that reads each golden's repo/revision/path, downloads the file from the Hub resolve URL (honouring HF_TOKEN for gated repos) into build/test-fixtures/tokenizer-goldens/<family>/, and checks the sha256. Point the test at that directory by default.
  3. For every golden and for both entry points — TokenizerFactory.fromTokenizerJsonString and GGUFTokenizer.fromTokenizerJson — assert either (a) every case's encode(text) == ids, or (b) the entry point throws UnsupportedTokenizerException (allowed only for fromTokenizerJsonString, see the fail-fast lane). Any other exception is a failure. The assertion message prints text, expected ids, expected token strings and actual ids.
  4. Support an optional per-case "expectedToFail": {"<entry point>": "<issue url>"} marker in the golden JSON: such a case must still fail, and the message tells the developer to remove the marker once it passes. This lets the test merge today and become the progress bar for the Lane 2 fixes and the engine lanes.
  5. Run ./gradlew :llm-core:downloadTokenizerGoldenFixtures :llm-core:jvmTest --tests '*TokenizerJsonGoldenParityTest*'; add the task to the smoke-reference.yml workflow's model-gated job if the maintainers agree (ask on the PR).

Acceptance

  • Test self-skips without fixtures; passes with fixtures and the current markers
  • Both entry points are exercised for every golden
  • Marker mechanism documented in the test's KDoc
  • Result reported back on #471

Notes

Gemma4TokenizerParityTest stays as is (it also checks atomic specials with hand-written ids); the new test is the general gate. Keep it in jvmTest: it needs the filesystem and network-fetched fixtures.

主要语言
Kotlin
星标
8
派生
3
平均合并
19 小时 45 分钟
30 天内合并 PR
38

环境准备

这个项目没有提供开发容器、Dockerfile 或贡献指南，环境需要你自己搭建：先看它的 README，通用步骤见我们的新手贡献指南。

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

SKaiNET-developers/SKaiNET-transformers 的其他 Issue

查看 SKaiNET-developers/SKaiNET-transformers 的全部 Issue

相似的 Issue

更多 Kotlin Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。