Should the stopword list be updated?
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 25/100
- Issue type
- Feature
- Clarity
- Needs clarification
- Activity status
- Stale
- Tech stack
- python
- Domain
- machine-learning
Research direction
Start with pattern/vector/stopwords-en.txt and review the comparison results in the linked Results directory. Read the find_important and compare functions in the StemmerChecker class, then determine whether the existing list should change and what updated contents would be justified.
Written by the indexing model from the issue text.
Description
I wrote a Python script to compare the list of stopwords currently used by pattern's vector module against other popular stopword lists to check whether an update is required.
A total of 11 sources of stopwords were used to compare with (listed below). For each source, the set of words present in the source but not in pattern's list of stopwords and vice versa were reported in the corresponding file. Comparison output was stored in the directory Results by my code here. File x.txt contains the comparison result by comparing the stopword list x against the stopword list used by pattern.
My code is available here: https://github.com/ni9elf/PatternClipsExperiments The comparison of stopword lists is performed by the find_important and compare function of the StemmerChecker class.
Lists of stopwords compared against:
-
unine.txt (formation retrieval multilingual resources from Universite de Neuchatel, Switzerland) link
-
princeton.txt (Algorithms book by Robert Sedgewick and Kevin Wayne, Princeton) link
-
nltk.txt (Natural Language Toolkit 3.2.5)
-
yoast.txt (YoastSEO is a text analysis and assessment library in JavaScript for SEO feedback) link
-
mysql.txt (MySQL Stopword list) link
-
ranksnl_short.txt (Default stopword list used by ranks.nl) link
-
ranksnl_long.txt (Longer version of stopword list used by ranks.nl) link
-
corenlp.txt (Stanford CoreNLP - natural language software) link
-
mallet.txt (MALLET (MAchine Learning for LanguagE Toolkit) from UMass Amherst) link
-
glasgow.txt (Information retrieval resources from University of Glasgow) link
-
onix.txt (Onix Text Retrieval Toolkit) link
- Dominant language
- Python
- Stars
- 8.9k
- Forks
- 1.6k
- PR merge metrics
- No merged PRs in 30d
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from clips/pattern
-
Difficulty 1/5 Under an hour Newbie friendliness 72/100
-
Difficulty 1/5 Under an hour Newbie friendliness 72/100
-
Difficulty 1/5 Under an hour Newbie friendliness 65/100
-
Difficulty 1/5 Under an hour Newbie friendliness 45/100
-
Difficulty 4/5 3-5 days Newbie friendliness 30/100
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
browser-use/browser-use#5905 ·
-
type: enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
ynput/ayon-python-api#363 ·
-
bug needs triage
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
modelscope/FunASR#3728 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
open-compass/opencompass#2655 ·