How to parse/tokenize only INDENT and DEDENT tokens?
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 35/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Stale
- Tech stack
- javascript, typescript
- Domain
- compilers, performance
Research direction
Start by reading the getAllTokens implementation and the parser/tokenizer entry points used by dt-python-parser. Benchmark tokenizing a 5,000-line Python file, focusing on the INDENT and DEDENT locations identified as token types 93 and 94; done means those locations are returned with substantially less waiting than getAllTokens.
Written by the indexing model from the issue text.
Description
Hello, I'm working on my VS Code extension, blockman, it renders blocks based on nested code blocks to make it easier to visually perceive the code.
video:
https://youtu.be/2Ajh8WQJvHs
The "dt-python-parser" package works really well, I use "getAllTokens" function to get all the tokens from python text file and then filter only type 93 (INDENT) and type 94 (DEDENT).
I need only INDENT and DEDENT locations, nothing more, but the "getAllTokens" function tokenizes everything and therefore losing much time, so if the file has more than 1000 lines, like 5000 lines or so, getAllTokens function takes many seconds to return the value, so rendering blocks take more time and waiting is not very comfortable for a user.
So, for the optimization, can I use the parser/tokenizer in such way that it would parse only INDENT/DEDENT locations and nothing more?
- Dominant language
- JavaScript
- Stars
- 25
- Forks
- 7
- PR merge metrics
- No merged PRs in 30d
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from DTStack/dt-python-parser
-
Difficulty 3/5 1-2 days Newbie friendliness 48/100
DTStack/dt-python-parser#28 ·
-
Difficulty 5/5 Over a week Newbie friendliness 20/100
DTStack/dt-python-parser#27 · 1 comment ·
-
Difficulty 3/5 1-2 days Newbie friendliness 45/100
DTStack/dt-python-parser#26 ·
-
Difficulty 3/5 1-2 days Newbie friendliness 45/100
DTStack/dt-python-parser#24 · 1 comment ·
-
Difficulty 5/5 Over a week Newbie friendliness 15/100
DTStack/dt-python-parser#15 · 3 comments ·
All issues in DTStack/dt-python-parser
Similar issues
-
bug customer-eng Durable Agents Inngest status: needs triage
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
-
optimization optimization:agents-md-curator
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
githubnext/gh-aw-cao#13475 ·
-
[BUG]: "Clear All" in Settings doesn't clear the saved analysis, old data comes back after reload Openbug
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
AOSSIE-Org/OrgExplorer#253 · 1 comment ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
oxc-project/oxc#26944 ·
-
ai-observability bug team/ai-observability
Difficulty 2/5 1-3 hours Newbie friendliness 78/100