Tokenizing a file with a lot of nested open arrays takes almost a minute
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 35/100
- Issue type
- Bug
- Clarity
- Needs clarification
- Activity status
- Stale
- Tech stack
- csharp
- Domain
- performance, tooling
Research direction
Reproduce tokenization using formats/json-tests/jvmTest/resources/spec_cases/n_structure_open_array_object.json and n_structure_100000_opening_arrays.json from the linked Kotlin repository, comparing them with the sub-millisecond files. Trace the tokenizer entry point to identify why deeply nested open arrays are slow; done means these files no longer take seconds to tokenize without changing tokenization results.
Written by the indexing model from the issue text.
Description
When tokenizing the files in the https://github.com/Kotlin/kotlinx.serialization repo, the cl100k_base tokenizer struggled on the following files:
- n_structure_open_array_object.json took 53.7s to tokenize
- n_structure_100000_opening_arrays.json took 6.9s to tokenize
While the rest of the files usually took less than a millisecond.
- Dominant language
- C#
- Stars
- 213
- Forks
- 37
- Avg merge
- 5d 1h
- Merged PRs (30d)
- 5
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from microsoft/Tokenizer
-
Difficulty 4/5 3-5 days Newbie friendliness 35/100
-
deepseek support? Open
Difficulty 5/5 Over a week Newbie friendliness 15/100
-
Difficulty 4/5 3-5 days Newbie friendliness 35/100
All issues in microsoft/Tokenizer
Similar issues
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
nightscout/nocturne#1425 ·
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
-
Documentation
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
cake-build/cake#5024 ·
-
Frontend status/draft TechnicalDebt
Difficulty 2/5 1-2 days Newbie friendliness 75/100
Altinn/altinn-auth#4143 ·