Need to add ignore null or empty text during korean text process

Open
#56 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
38/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Stale
Tech stack
python
Domain
data-engineering

Research direction

No files or tests are named. Locate the Korean text-processing data-loading entry point and follow the RDD/DataFrame pipeline; inspect how null and empty strings are represented, then run the existing processing tests or a representative dataset. Done means both null and empty text are excluded after loading.

Written by the indexing model from the issue text.

Description

bug
Agenda
  • Some raw datasets can have empty or null text.
  • Using filter method in RDD or DF, text like "" need to be ignored during this process.
How to solve this problem?
  • Add filter method after load the data
  • Add condition if the text data is null or empty in filter method
Dominant language
Python
Stars
93
Forks
26
No merged PRs in 30d

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from EleutherAI/dps

All issues in EleutherAI/dps

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.