[Lane 6 · review] tokenizer.json parity — DARC review
维护者通常 1 天内回复
还没有人认领这个 Issue。
评估
- 难度
- 4/5
- 预计耗时
- 1-2 天
- 新手友好度
- 18/100
- Issue 类型
- 功能
- 描述清晰度
- 描述清楚
- 活跃度
- 活跃
调研方向
Read parent issue #1321 and compare each acceptance criterion with the shipped tokenizer code and docs. Regenerate goldens with the pinned tokenizers version using the Lane 4 script, then run the listed Gradle fixture and test tasks and hand-check three cases in Python. Verify the cited semantics and the specified failure behavior, added-token vocabSize, decode fixture, and changelog; report pass or fail on #1321 and update or close #912 as appropriate.
由索引模型根据 Issue 内容生成。
描述
Sub-issue of #1321 (HF tokenizer.json parity for BPE encoders).
Lane: 6 · Review
Skill needed: DARC review — must not have implemented any Lane 2 or Lane 4 task. Kotlin reading skills and access to a Python environment to re-run the goldens.
Size: s (a few hours)
Blocked by: all Lane 2 and Lane 4 sub-issues merged.
What to do
- D — Document. Re-read the parent's problem statement against the shipped code: are the four gaps closed, or did any lane narrow its scope? Record deviations as comments on the parent.
- A — Assess. Regenerate the goldens with the pinned
tokenizersversion (Lane 4 script) and run./gradlew :skainet-io:skainet-io-core:downloadTokenizerJsonGoldenFixtures :skainet-io:skainet-io-core:jvmTest :skainet-io:skainet-io-core:linuxX64Test. Spot-check three cases by hand with the Python library. - R — Research. Click every link in the parent, the docs page and the PR descriptions; confirm the
tokenizerssemantics cited (added-token matching, merges form) still hold at the pinned version. - C — Code. Check that unsupported normalizer / pre-tokenizer types fail loudly, that
vocabSizeincludes added tokens, that the #912 decode fixture is present, and thatCHANGELOG.mdlists the behaviour changes. - Close #912 if its four asks are met, or comment on it with what remains.
Acceptance
- Written review comment on #1321 with a pass/fail per acceptance criterion of the parent
- Docs support matrix matches the code
- #912 closed or updated
- Result reported back on #1321 (then the parent can be closed)
Notes
@DarcValidated targets TensorOps functions and does not apply to the tokenizer package; the review comment on the parent is the durable record here.
- 主要语言
- Kotlin
- 星标
- 52
- 派生
- 15
- 平均合并
- 1 天 15 小时
- 30 天内合并 PR
- 36
环境准备
- 没有 Dockerfile 或 Docker Compose 文件
- 没有 Pull Request 模板
- 阅读贡献指南
从这里开始
- 先读完整个 Issue，再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库，在一个分支上完成修改。
- 提交 Pull Request，并在描述里引用这个 Issue 编号。
SKaiNET-developers/SKaiNET 的其他 Issue
-
coding good first issue size:xs skill:kotlin-core sub-issue
难度 2/5 1-3 小时 新手友好度 88/100
SKaiNET-developers/SKaiNET#1323 ·
维护者通常 1 天内回复
-
coding good first issue platform size:xs skill:js sub-issue
难度 2/5 1-3 小时 新手友好度 72/100
SKaiNET-developers/SKaiNET#1232 ·
维护者通常 1 天内回复
-
skeep tracking
难度 5/5 一周以上 新手友好度 15/100
SKaiNET-developers/SKaiNET#1331 ·
维护者通常 1 天内回复
-
documentation good first issue size:s skill:docs sub-issue
难度 3/5 1-2 天 新手友好度 78/100
SKaiNET-developers/SKaiNET#1329 ·
维护者通常 1 天内回复
-
coding good first issue size:s skill:kotlin-core sub-issue
难度 3/5 半天 新手友好度 75/100
SKaiNET-developers/SKaiNET#1328 ·
维护者通常 1 天内回复
查看 SKaiNET-developers/SKaiNET 的全部 Issue
相似的 Issue
-
afk-ok area:data-quality importer size:S
难度 2/5 1-3 小时 新手友好度 77/100
enorm-labs/event-junkie#3166 ·
维护者通常 1 天内回复
-
bugbane/qf good first issue
难度 2/5 1-3 小时 新手友好度 64/100
osservatorionessuno/bugbane#173 ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 78/100
-
documentation
难度 2/5 1-3 小时 新手友好度 66/100
theopenconversationkit/tock#2113 ·
维护者通常 1 天内回复
-
Bayern feature:Application ready web
难度 2/5 1-3 小时 新手友好度 67/100
digitalfabrik/entitlementcard#3176 ·
维护者通常 7 天内回复