Deserialization for PCA model seems to be broken. Failed to find a default value for "k"

Open
  • Is something wrong?
#831 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
35/100
Issue type
Bug
Clarity
Needs clarification
Activity status
Stale
Tech stack
python, scala, spark
Domain
machine-learning

Research direction

Start by tracing PCA model parameter handling in PCA.scala and default lookup in params.scala, focusing on how k is retained through serializeToBundle and deserializeFromBundle. Reproduce the reported Python PCAModel case and verify that the deserialized model retains k and can transform without the NoSuchElementException.

Written by the indexing model from the issue text.

Description

I have created a PCA model and able to serializeToBundle without any issues. However, after deserializeFromBundle and trying to transform, I am encountering below error:

py4j.protocol.Py4JJavaError: An error occurred while calling o278.transform.
: java.util.NoSuchElementException: Failed to find a default value for k
at org.apache.spark.ml.param.Params.$anonfun$getOrDefault$2(params.scala:756)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.ml.param.Params.getOrDefault(params.scala:756)
at org.apache.spark.ml.param.Params.getOrDefault$(params.scala:753)
at org.apache.spark.ml.PipelineStage.getOrDefault(Pipeline.scala:41)
at org.apache.spark.ml.param.Params.$(params.scala:762)
at org.apache.spark.ml.param.Params.$$(params.scala:762)
at org.apache.spark.ml.PipelineStage.$(Pipeline.scala:41)
at org.apache.spark.ml.feature.PCAParams.validateAndTransformSchema(PCA.scala:55)
at org.apache.spark.ml.feature.PCAParams.validateAndTransformSchema$(PCA.scala:51)
at org.apache.spark.ml.feature.PCAModel.validateAndTransformSchema(PCA.scala:122)
at org.apache.spark.ml.feature.PCAModel.transformSchema(PCA.scala:156)
at org.apache.spark.ml.PipelineStage.transformSchema(Pipeline.scala:71)
at org.apache.spark.ml.feature.PCAModel.transform(PCA.scala:146)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:282)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:238)
at java.lang.Thread.run(Thread.java:750)

However, I have passed value of k while initialization the model and serializing it.

model = PCAModel(k=3, inputCol="features", outputCol="pca")

It seems to be very similar to the issue: https://github.com/combust/mleap/issues/481 where certain variables are missed to be set.

Dominant language
Scala
Stars
1.5k
Forks
313
Avg merge
22m
Merged PRs (30d)
1

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

  • No Dockerfile or Docker Compose file
  • No pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from combust/mleap

All issues in combust/mleap

Similar issues

More Scala issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.