[Lane 4 · kotlin-core] tokenizer.json parity — fixture-gated jvmTest replaying the goldens through TokenizerFactory.fromTokenizerJson

Fermée
  • Quelque chose ne va pas ?
#1,328 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub

Les mainteneurs répondent en général sous 1 jour

Personne n'a encore pris cette issue.

Évaluation

Difficulté
3/5
Temps estimé
Une demi-journée
Accessibilité débutants
75/100
Type d'issue
Fonctionnalité
Clarté
Clairement spécifiée
Activité
Active
Stack technique
kotlin
Domaine
machine-learning, testing

Piste de recherche

Start by reading QwenByteLevelBpeTokenizerFixtureTest.kt and the downloadQwenTokenizerFixtures task in skainet-io-core/build.gradle.kts; the issue gives both as templates. Add TokenizerJsonGoldenParityTest.kt and a task that downloads and verifies the two golden fixtures, then run the listed Gradle command. Done means the test skips when fixtures are absent and, with fixtures present, checks vocabulary size and encoded IDs while reporting text, expected IDs and tokens, and actual IDs.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Description

coding good first issue size:s skill:kotlin-core sub-issue

Sub-issue of #1321 (HF tokenizer.json parity for BPE encoders).

Lane: 4 · Ground truth / CI
Skill needed: Kotlin and Gradle. You will copy an existing test file and an existing download task; no tokenizer internals needed.
Size: s (a few hours)
Blocked by: the golden-generation lane (needs the JSON files). Can be merged before the Kotlin fixes — it is allowed to fail on the not-yet-fixed rows if you mark them, see Notes.

What to do

  1. Copy QwenByteLevelBpeTokenizerFixtureTest.kt to TokenizerJsonGoldenParityTest.kt in the same jvmTest package. Keep the "skip with a printed notice when the fixture is absent" behaviour.
  2. Copy the downloadQwenTokenizerFixtures task in skainet-io-core/build.gradle.kts#L148-L153 into a downloadTokenizerJsonGoldenFixtures task that fetches the two Laya tokenizer.json files into build/test-fixtures/tokenizer-goldens/<name>/tokenizer.json from the Hub resolve URL at the pinned revision, and verifies the SHA-256 stored in the golden JSON.
  3. In the test, for each golden JSON in src/jvmTest/resources/tokenizer-goldens/: load the matching tokenizer.json through sk.ainet.io.tokenizer.TokenizerFactory.fromTokenizerJson, assert vocabSize == vocab_size_with_added, then for every case assert encode(text).toList() == ids with a message that prints the text, the expected and the actual ids and the expected token strings (the tokens array) — that is what makes a failure diagnosable without Python.
  4. Parse the golden JSON with kotlinx.serialization.json (already a dependency of the module).
  5. Run ./gradlew :skainet-io:skainet-io-core:downloadTokenizerJsonGoldenFixtures :skainet-io:skainet-io-core:jvmTest --tests '*TokenizerJsonGoldenParityTest*'.

Acceptance

  • Test self-skips when the fixtures are absent (CI without network stays green)
  • With fixtures present and all sibling lanes merged, every case in both goldens passes
  • Failure output shows text, expected ids + tokens, actual ids
  • Result reported back on #1321

Notes

To merge this before the fixes land, support an optional "expectedToFail": "<issue url>" field per case in the golden JSON and turn those cases into "assert it still fails, otherwise tell the developer to remove the marker" — the same canary pattern #912 describes. Then each fix lane removes its markers in its own PR, and the test is the progress bar.

Langage dominant
Kotlin
Étoiles
52
Forks
15
Merge moyen
23 h 43 min
PR mergées (30 j)
82

Préparer son environnement

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Autres issues de SKaiNET-developers/SKaiNET

Toutes les issues de SKaiNET-developers/SKaiNET

Issues similaires

Plus d'issues Kotlin

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.