microsoft/SynapseML

Featurizer should provide option to pass through missing values as Double.NaN instead of removing rows (currently the default)

Open

#304 opened on May 18, 2018

View on GitHub
 (5 comments) (0 reactions) (1 assignee)Scala (5,228 stars) (861 forks)batch import
enhancementgood first issuehelp wanted

Description

Hi! Using lightGBM I faced another problem. I'm not sure if it is bug or feature :) but in our data we have a lot of empty values, so before we used sparse vector to store features, and it worked fine with our previous lib. But when i tried to use featurizer, that you provide - i mentioned, that you skip all raws if any nulls are presents as a feature. you can see it in example in attachment. So is it possible to have sparse feature vector for lightGBM training?

https://gist.github.com/ekaterina-sereda-rf/929183b9bcbbf5baf15eec3e81329992

Contributor guide

Tech stack
scala
Domain
machine learningdata
Issue type
feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-3 hours
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
clear
Prerequisites
basic ScalaSpark ML concepts
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
40
Research direction
Examine the Featurizer implementation in the SynapseML codebase, likely under src/main/scala. The current behavior drops rows with null features; the issue requests an option to retain missing values as Double.NaN. Look for the Featurizer class or object, and identify where the null handling logic is. The change involves adding a configuration parameter (e.g., setHandleMissing) and modifying the transform logic to either drop rows or pass NaN. Since the issue has an assignee and is old, it's advisable to check if any related PRs exist or if the maintainers have provided guidance in the comments.