Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

[Lane 4 · kotlin-core] tokenizer.json parity — gated parity test replaying the goldens through both tokenizer.json entry points

Aperta
  • Qualcosa non va?
#477 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
1-2 giorni
Idoneità per principianti
64/100
Tipo di issue
Funzionalità
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
kotlin
Ambito
testing

Direzione di ricerca

Read Gemma4TokenizerParityTest.kt for the self-skip pattern and llm-core/build.gradle.kts for the Gradle task entry point. Add TokenizerJsonGoldenParityTest.kt under llm-core/src/jvmTest/.../tokenizer/, then run the fixture download task and targeted jvmTest command from the issue. Done means both tokenizer JSON entry points are checked against every golden, expected failures are enforced, and the test skips with a notice when fixtures are unavailable.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

coding good first issue size:s skill:kotlin-core sub-issue

Sub-issue of #471 (tokenizer.json parity gate).

Lane: 4 · Ground truth / CI
Skill needed: Kotlin and Gradle; you copy an existing self-skipping test. No tokenizer internals.
Size: s (a few hours)
Blocked by: the golden-generation lane (needs the JSON files)

What to do

  1. Copy the self-skip pattern of Gemma4TokenizerParityTest.kt into a new llm-core/src/jvmTest/kotlin/sk/ainet/apps/llm/tokenizer/TokenizerJsonGoldenParityTest.kt.
  2. Fixture location: $TOKENIZER_GOLDEN_FIXTURES_DIR/<family>/tokenizer.json (environment variable; skip with a printed notice when unset). Add a Gradle task downloadTokenizerGoldenFixtures in llm-core/build.gradle.kts that reads each golden's repo/revision/path, downloads the file from the Hub resolve URL (honouring HF_TOKEN for gated repos) into build/test-fixtures/tokenizer-goldens/<family>/, and checks the sha256. Point the test at that directory by default.
  3. For every golden and for both entry points — TokenizerFactory.fromTokenizerJsonString and GGUFTokenizer.fromTokenizerJson — assert either (a) every case's encode(text) == ids, or (b) the entry point throws UnsupportedTokenizerException (allowed only for fromTokenizerJsonString, see the fail-fast lane). Any other exception is a failure. The assertion message prints text, expected ids, expected token strings and actual ids.
  4. Support an optional per-case "expectedToFail": {"<entry point>": "<issue url>"} marker in the golden JSON: such a case must still fail, and the message tells the developer to remove the marker once it passes. This lets the test merge today and become the progress bar for the Lane 2 fixes and the engine lanes.
  5. Run ./gradlew :llm-core:downloadTokenizerGoldenFixtures :llm-core:jvmTest --tests '*TokenizerJsonGoldenParityTest*'; add the task to the smoke-reference.yml workflow's model-gated job if the maintainers agree (ask on the PR).

Acceptance

  • Test self-skips without fixtures; passes with fixtures and the current markers
  • Both entry points are exercised for every golden
  • Marker mechanism documented in the test's KDoc
  • Result reported back on #471

Notes

Gemma4TokenizerParityTest stays as is (it also checks atomic specials with hand-written ids); the new test is the general gate. Keep it in jvmTest: it needs the filesystem and network-fetched fixtures.

Lingua principale
Kotlin
Stelle
8
Fork
3
Merge medio
1g 3h
PR unite (30g)
36

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di SKaiNET-developers/SKaiNET-transformers

Tutte le issue di SKaiNET-developers/SKaiNET-transformers

Issue simili

Altre issue su Kotlin

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.