rhiever/datacleaner

Replace +/- Infs with Max/Min

Open

#11 opened on May 22, 2017

View on GitHub
 (1 comment) (0 reactions) (0 assignees)Python (1,040 stars) (211 forks)batch import
enhancementhelp wanted

Description

Hi there,

datacleaner seems quite interesting. Cleaning Data is always annoying and tools are missing.

If I have seen it right, you impute NaNs. You could also consider to replace +/- Infs by Max/Min of the respective column.

We have implemented that In the tsfresh impute function. Maybe you can use some of the code there.

Contributor guide

Tech stack
pythonpandas
Domain
data
Issue type
feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-3 hours
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
clear
Prerequisites
Basic pandasUnderstanding of imputation
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
60
Research direction
Look at the current imputation code in the repository, likely in a module like impute.py. Reference the tsfresh implementation at https://github.com/blue yonder/tsfresh/blob/ca0fdbf7c1f3f165b36f981dd59bd7dc34af4113/tsfresh/utilities/dataframe functions.py for inspiration. Implement logic to detect infinities and replace them with column min/max, ensuring compatibility with existing NaN imputation.