Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

[Lane 6 · review] tokenizer.json parity — DARC review

Aperta
  • Qualcosa non va?
#1,330 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
1-2 giorni
Idoneità per principianti
18/100
Tipo di issue
Funzionalità
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
kotlin, python
Ambito
machine-learning, testing-qa

Direzione di ricerca

Read parent issue #1321 and compare each acceptance criterion with the shipped tokenizer code and docs. Regenerate goldens with the pinned tokenizers version using the Lane 4 script, then run the listed Gradle fixture and test tasks and hand-check three cases in Python. Verify the cited semantics and the specified failure behavior, added-token vocabSize, decode fixture, and changelog; report pass or fail on #1321 and update or close #912 as appropriate.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

assessment size:s skill:review sub-issue

Sub-issue of #1321 (HF tokenizer.json parity for BPE encoders).

Lane: 6 · Review
Skill needed: DARC review — must not have implemented any Lane 2 or Lane 4 task. Kotlin reading skills and access to a Python environment to re-run the goldens.
Size: s (a few hours)
Blocked by: all Lane 2 and Lane 4 sub-issues merged.

What to do

  1. D — Document. Re-read the parent's problem statement against the shipped code: are the four gaps closed, or did any lane narrow its scope? Record deviations as comments on the parent.
  2. A — Assess. Regenerate the goldens with the pinned tokenizers version (Lane 4 script) and run ./gradlew :skainet-io:skainet-io-core:downloadTokenizerJsonGoldenFixtures :skainet-io:skainet-io-core:jvmTest :skainet-io:skainet-io-core:linuxX64Test. Spot-check three cases by hand with the Python library.
  3. R — Research. Click every link in the parent, the docs page and the PR descriptions; confirm the tokenizers semantics cited (added-token matching, merges form) still hold at the pinned version.
  4. C — Code. Check that unsupported normalizer / pre-tokenizer types fail loudly, that vocabSize includes added tokens, that the #912 decode fixture is present, and that CHANGELOG.md lists the behaviour changes.
  5. Close #912 if its four asks are met, or comment on it with what remains.

Acceptance

  • Written review comment on #1321 with a pass/fail per acceptance criterion of the parent
  • Docs support matrix matches the code
  • #912 closed or updated
  • Result reported back on #1321 (then the parent can be closed)

Notes

@DarcValidated targets TensorOps functions and does not apply to the tokenizer package; the review comment on the parent is the durable record here.

Lingua principale
Kotlin
Stelle
52
Fork
15
Merge medio
1g 4h
PR unite (30g)
37

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di SKaiNET-developers/SKaiNET

Tutte le issue di SKaiNET-developers/SKaiNET

Issue simili

Altre issue su Kotlin

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.