Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

[Lane 5 · docs] tokenizer.json parity — reference page: tokenizer.json support matrix

Aperta
  • Qualcosa non va?
#1,329 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
3/5
Tempo stimato
1-2 giorni
Idoneità per principianti
78/100
Tipo di issue
Documentazione
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
kotlin
Ambito
ai, documentation

Direzione di ricerca

Read TokenizerFactory.kt and the Lane 2 measurements from #1321 to fill the support matrix, then follow an existing reference page for its Antora structure. Add tokenizer-json-support.adoc, register it in nav.adoc, and add the KDoc pointer; check the Lane 4 sub-issues for the Python golden script and gated test. Done when the page is listed under Reference, unsupported or partial rows link to open issues, and the result is reported on #1321.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

documentation good first issue size:s skill:docs sub-issue

Sub-issue of #1321 (HF tokenizer.json parity for BPE encoders).

Lane: 5 · Docs
Skill needed: AsciiDoc and the ability to read a Kotlin when statement. No build or tokenizer knowledge.
Size: s (a few hours)
Blocked by: nothing for the first version; update the table as the Lane 2 PRs merge.

What to do

  1. The engine docs have no tokenizer page. Add docs/modules/ROOT/pages/reference/tokenizer-json-support.adoc and register it in docs/modules/ROOT/nav.adoc under Reference.
  2. Content: a support matrix of what sk.ainet.io.tokenizer.TokenizerFactory accepts, one row per tokenizer.json feature — model.type (BPE / Unigram / WordPiece), model.merges form (string / array), byte_fallback, ignore_merges, each normalizer type, each pre_tokenizer type, added_tokens (special: true / false, the four flags), post_processor (not applied — callers add BOS/EOS themselves, say so), decoder. Columns: Supported (yes / no / partial), Since version, Issue link. Derive the current state from TokenizerFactory.kt and the #1321 measurements; link each "no" to its lane sub-issue.
  3. A second short section "GGUF vs tokenizer.json": the factory dispatches per architecture, not per file format (the KDoc at the top of TokenizerFactory.kt already says this well — reuse it).
  4. A "How to verify parity" paragraph pointing to the Python golden script and the gated test from the Lane 4 sub-issues.
  5. Add a one-line pointer from the TokenizerFactory KDoc to the new page. Build the docs locally if you can (docs/README or the Antora playbook in the repo explains how); otherwise a careful preview in the GitHub editor is acceptable for a first PR.

Acceptance

  • Page renders in the Antora nav under Reference
  • Every "no" / "partial" row links to an open issue
  • fromTokenizerJson KDoc links to the page
  • Result reported back on #1321

Notes

Follow the structure of an existing reference page in docs/modules/ROOT/pages/reference/ for headings and the :description: attribute. The downstream repo (SKaiNET-transformers) has explanation/tokenizer-internals.adoc; do not duplicate it, link to it for the algorithm descriptions.

Lingua principale
Kotlin
Stelle
52
Fork
15
Merge medio
1g 8h
PR unite (30g)
37

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di SKaiNET-developers/SKaiNET

Tutte le issue di SKaiNET-developers/SKaiNET

Issue simili

Altre issue su Kotlin

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.