Rework Search engine: stemming and ranking
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 35/100
Research direction
The issue lays out goals for stemming and search ranking but names no files, tests, or implementation entry points. Start by locating the search and stemming code and its existing tests, then compare candidate stemmers against the relevance examples in the issue. Done means retrieval avoids both missing and diluted results, and exact matches always rank above stemmed variations.
Written by the indexing model from the issue text.
Description
in this ticket I'll state the objective and proposal several candidates and compare them with previous work
How Muayyad Alsadi (me) define stemming differently
it's not a linguistic morphology that returns a proper dictionary word without prefixes and suffixes
it's a hashing function that its job is to cluster relevant more terms together and more irrelevant terms apart
for example, if stem('using')=stem('us') this is bad despite us being a dictionary term, because us (united states) is not relevant to "using"
on the other hand stemmer that return non word let's say stem(classes) = stem(class) = clas is a good stemmer despite clas with single s being a non dictionary proper term
Stemer choice
How to report a good stemming case
stem(A) = stem(B)when A and B are relevantstem(A) != stem (B)when A and B are irrelevant
How to report a bad stemming case
stem(A) = stem(B)when A and B are irrelevantstem(A) != stem (B)when A and B are relevant
Search Engine Success
Two fronts:
- retrieval (what is in the matched set, regardless of order), a search that returns too few is bad (missing), too many (or all) is bad too (dilution)
- ranking: given the same result set, how to score them, which one goes first
Stemmed term should never beat Exact regardless of stemmer
the formula should be designed so that regardless of how bad the stemmer is and how bad the choice of the blend multipliers, exact should always win.
a simple trick is to put max on denominator so that a stemmed variation of a term can never beat any of the exact.
- Dominant language
- C++
- Stars
- 2.5k
- Forks
- 158
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 52
Getting set up
- No Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from redhat-et/ripwire
-
Difficulty 2/5 Under an hour Newbie friendliness 78/100
redhat-et/ripwire#334 · 7 comments ·
Maintainers usually reply within 1 day
-
help wanted
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
redhat-et/ripwire#68 · 2 comments ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 85/100
Maintainers usually reply within 1 day
-
help wanted
Difficulty 5/5 Over a week Newbie friendliness 25/100
Maintainers usually reply within 1 day
-
Difficulty 5/5 Over a week Newbie friendliness 15/100
redhat-et/ripwire#409 · 10 comments ·
Maintainers usually reply within 1 day
All issues in redhat-et/ripwire
Similar issues
-
bug IO
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
MrNeRF/LichtFeld-Studio#3824 ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 88/100
-
bug code-review testing tooling
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Maintainers usually reply within 1 day
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
aristocratos/btop#1945 ·
Maintainers usually reply within 2 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 79/100
boykopovar/AnyPS5#11291 ·
Maintainers usually reply within 1 day