Unexpected tokenization
Nobody has claimed this yet.
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 42/100
Research direction
Start by reproducing the en.tokenize example from the issue and locate the tokenizer implementation and its related tests. Compare the current token list with the expected two-item list; done means the example produces the expected grouping without breaking existing tokenization behavior.
Written by the indexing model from the issue text.
Description
Hello,
This is a peculiar case of enumeration where the tokenization gives inappropriate results.
en.tokenize('See Section 3.) Or Section 2.)') results in ['See Section 3 .', ') Or Section 2 .', ')'] while the expectation is ['See Section 3 . )', 'Or Section 2 . )']
- Dominant language
- Python
- Stars
- 8.9k
- Forks
- 1.6k
- PR merge metrics
- No merged PRs in 30d
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from clips/pattern
-
Difficulty 1/5 Under an hour Newbie friendliness 72/100
-
Difficulty 1/5 Under an hour Newbie friendliness 72/100
-
Difficulty 1/5 Under an hour Newbie friendliness 65/100
-
Difficulty 1/5 Under an hour Newbie friendliness 45/100
-
Difficulty 4/5 3-5 days Newbie friendliness 30/100
Similar issues
-
documentation help wanted
Difficulty 2/5 1-3 hours Newbie friendliness 90/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 90/100
simonw/sqlite-utils#872 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100