Better support for overlapping scripts (eg. CJK) in the segmenter
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 48/100
Research direction
Start by reading the segmenter documentation's "Multiple Frequency Data Sets and Merge Groups" section, then locate the segmenter implementation it describes. Trace how merging currently handles codepoints unique to one script and determine how overlapping scripts such as CJK should be segmented; the work is done when those overlapping-script cases are supported as specified by the documentation.
Written by the indexing model from the issue text.
Description
In the segmenter within a particular script merging is only performed for codepoints that are unique to that script. We need to extend the segmenter to be able to handle segmentations when scripts with large overlaps are provided. The common example of this CJK. The segmenter documentation discusses this in more detail and proposes a solution, see the "Multiple Frequency Data Sets and Merge Groups" section.
- Dominant language
- C++
- Stars
- 13
- Forks
- 9
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 7
Getting set up
- No Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from w3c/ift-encoder
-
Difficulty 5/5 Over a week Newbie friendliness 30/100
w3c/ift-encoder#271 ·
-
Consider adding a config setting that exempts high frequency codepoints from minimum group sizesOpen
Difficulty 5/5 Over a week Newbie friendliness 42/100
w3c/ift-encoder#269 ·
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
w3c/ift-encoder#262 ·
-
latest bazel does not workMay be free again @garretrieger claimed this 194 days ago, and no pull request is open. Open
w3c/ift-encoder#202 · 1 comment · 1 assignee ·
-
Difficulty 4/5 3-5 days Newbie friendliness 38/100
w3c/ift-encoder#108 · 1 comment ·
Similar issues
-
needs-format
Difficulty 2/5 1-3 hours Newbie friendliness 69/100
Niko1221/Strata#3368 · 1 comment ·
Maintainers usually reply within 3 days
-
bug
Difficulty 1/5 1-3 hours Newbie friendliness 76/100
dragonflydb/dragonfly#8617 · 1 reaction ·
Maintainers usually reply within 1 day
-
bug needs triage
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
project-chip/connectedhomeip#74842 ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 88/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 67/100
smartmontools/smartmontools#670 ·
Maintainers usually reply within 1 day