Improve stop word handling - customizable and/or multi-language
#7,330 创建于 2025年6月6日
仓库指标
- 星标
- (10,056 个星标)
- PR 合并指标
- (平均合并 4天 18小时) (30 天内合并 10 个 PR)
描述
Currently we have a single, fixed, English-only list of stop words. https://github.com/OpenRefine/OpenRefine/blob/89a9e8d5bd8a97374e6a07bdfd1ff960169a78e2/modules/core/src/main/java/com/google/refine/model/recon/StandardReconConfig.java#L744-L752
It would be desirable to improve this so that the list is some combination of:
- visible
- language specific, for languages that we support
- user customizable
Currently this list is only used as part of the reconciliation process, but if we had a robust list of stop words, it could potentially be used other places (fingerprint cf #3200 , etc)
Proposed solution
- Make the current list of stop words editable via preferences with a preference key of
stopwordsand an initial value of "the,a,and,of,on,in,at,by" and modify StandardReconConfig to fetch the preference value, split on commas, trim any whitespace on each piece (to make user editing less error prone) and add them to the stop words set.
Alternatives considered
- Use an expanded list of English stop words as the starting point and make it editable via settings
- Find lists of stop words for N top languages and make them available as starting templates (note that the user can also do this on their own with either of the above solutions)
Lists useful for the last two include:
- English from NLTK - https://gist.github.com/sebleier/554280 although this needs to be filtered to remove tokens which aren't stop word related under our whitespace tokenizer (e.g. "won" can get tokenized from "won't" using the NLTK tokenizer, but not OpenRefine's)
- multilingual - https://github.com/stopwords-iso/stopwords-iso/blob/master/stopwords-iso.json
Additional context
Since we don't really know the language of the text that the user is operating on, it's probably dangerous to assume that it's the same as the UI language, so a better approach might be to have one or more "standard" lists of stop words that the user can use as a starting point and then customize. Even a single customizable list per user would be an improvement on the current situation.
If we support stop word lists for multiple languages, we'll need a way to choose which one to use. This might be based on knowing the language of the user, project, or column, or it might be something that the user specifies.