[ja] `.filter` is used instead of `.map` for non-filter methods
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 45/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Stale
- Tech stack
- python, spark
- Domain
- data-engineering
Research direction
Read dps/spark/jobs/japanese_job.py around lines 64-75 and inspect preprocess_text, reduce_japanese_emoticon, and remove_symbols in dps/spark/prep/japanese_prep.py. Check the surrounding RDD transformations and confirm that boolean *_filter methods remain filters. Done means the three preprocessing calls transform records and the actual filter methods still filter them.
Written by the indexing model from the issue text.
Description
On https://github.com/EleutherAI/dps/blob/bec4078f341037879feab1d5c82668745b28aa55/dps/spark/jobs/japanese_job.py#L64-L75 there are several cases where we are using .filter but instead it should be a .map.
For example https://github.com/EleutherAI/dps/blob/bec4078f341037879feab1d5c82668745b28aa55/dps/spark/jobs/japanese_job.py#L73
calls
https://github.com/EleutherAI/dps/blob/bec4078f341037879feab1d5c82668745b28aa55/dps/spark/prep/japanese_prep.py#L64-L67
but in effect this is doing nothing because the expression within .filter is always is true, as long as text is non-empty:
>>> def reduce_japanese_emoticon(text):
... text = re.sub("w{3,}", "www", text)
... text = re.sub("笑{2,}", "笑", text)
... return text
>>> rdd = sc.parallelize([{'text': 'wwwwasdf'}, {'text': '1234笑笑笑'}, {'text': ''}])
>>> rdd.filter(lambda x: reduce_japanese_emoticon(x['text'])).collect()
[{'text': 'wwwwasdf'}, {'text': '1234笑笑笑'}]
Thus, I think the following cases of .filter are simply doing nothing instead of the intended preprocessing:
preprocess_texton https://github.com/EleutherAI/dps/blob/bec4078f341037879feab1d5c82668745b28aa55/dps/spark/jobs/japanese_job.py#L70reduce_japanese_emoticonon https://github.com/EleutherAI/dps/blob/bec4078f341037879feab1d5c82668745b28aa55/dps/spark/jobs/japanese_job.py#L73remove_symbolson https://github.com/EleutherAI/dps/blob/bec4078f341037879feab1d5c82668745b28aa55/dps/spark/jobs/japanese_job.py#L75
The remaining calls to methods that end with _filter (e.g. japanese_bad_words_filter, doc_len_filter, etc.) are actually filter methods that return booleans so they should be OK.
- Dominant language
- Python
- Stars
- 93
- Forks
- 26
- PR merge metrics
- No merged PRs in 30d
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from EleutherAI/dps
-
k Open
Difficulty 5/5 Over a week Newbie friendliness 10/100
EleutherAI/dps#82 ·
-
Difficulty 4/5 3-5 days Newbie friendliness 20/100
EleutherAI/dps#81 · 3 comments ·
-
Difficulty 4/5 3-5 days Newbie friendliness 25/100
EleutherAI/dps#65 · 1 comment ·
-
EleutherAI/dps#62 · 1 assignee ·
-
enhancement
Difficulty 5/5 Over a week Newbie friendliness 15/100
EleutherAI/dps#57 ·
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100