Input schema of a StringIndexed Column is always string
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 35/100
- Issue type
- Bug
- Clarity
- Mostly clear
- Activity status
- Stale
- Tech stack
- scala, spark
- Domain
- machine-learning
Research direction
Start at mleap-core/src/main/scala/ml/combust/mleap/core/feature/StringIndexerModel.scala line 50 and trace how the exported transformer builds its inputSchema. Compare the Spark StringIndexer behavior for numeric and string inputs, then determine whether the original datatype is available in the MLeap bundle and can be preserved in transformer.inputSchema.
Written by the indexing model from the issue text.
Description
I have a dataset with a numerical column rank with values ranging from 1 to 5. When this dataset is read in spark with inferSchema = true, the dataType of the column is inferred as int by spark. A model is built by applying StringIndexer on the rank column as one of the stages and the org.apache.spark.ml.PipelineModel is exported as an mleap bundle. When we read the ml.combust.mleap.runtime.frame.Transformer from the mleap bundle and observe the transformer.inputSchema, it returns the dataType of the rank column as String instead of an int.
Looks like this is because of
https://github.com/combust/mleap/blob/master/mleap-core/src/main/scala/ml/combust/mleap/core/feature/StringIndexerModel.scala#L50
According to spark docs for StringIndexer,
If the input column is numeric, we cast it to string and index the string values.
so, an input column for a string indexer stage can be a numeric or string type.
Is there a way to pass the information of the actual datatype of the column to the transformer, so that transformer.inputSchema can return same instead of String for all the cases?
- Dominant language
- Scala
- Stars
- 1.5k
- Forks
- 313
- Avg merge
- 22m
- Merged PRs (30d)
- 1
Getting set up
Starts the project's dev container in your browser, under your own GitHub account.
- No Dockerfile or Docker Compose file
- No pull request template
- No contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from combust/mleap
-
[Security] Incomplete fix for CVE-2023-5245: zip slip via directory entries in FileUtil.extractOpen
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
Difficulty 5/5 Over a week Newbie friendliness 15/100
-
Question: Alternative to MLeap for Real-Time Inference Without Spark Context with SparkXGBClassifierOpen
Difficulty 5/5 Over a week Newbie friendliness 20/100
-
Difficulty 5/5 Over a week Newbie friendliness 15/100
-
Difficulty 4/5 3-5 days Newbie friendliness 32/100
Similar issues
-
module: backend type: bug/reported
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
OpenXiangShan/XiangShan#6938 ·
Maintainers usually reply within 3 days
-
documentation good first issue
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
Babali42/DrumBeatRepo#639 ·
Maintainers usually reply within 1 day
-
module: core presentation conversion
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
bigbluebutton/bigbluebutton#26508 ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 84/100
Maintainers usually reply within 1 day
-
Difficulty 1/5 1-3 hours Newbie friendliness 82/100
Maintainers usually reply within 3 days