combust/mleap

SQLTransformer

Open

#126 opened on Feb 13, 2017

View on GitHub
 (20 comments) (0 reactions) (0 assignees)Scala (1,461 stars) (312 forks)batch import
help wanted

Description

We have a Spark ML pipeline with SQLTransformer along with UDF.

I checked at https://github.com/combust/mleap/tree/master/mleap-spark/src/main/scala/org/apache/spark/ml/bundle/ops/feature and didn't find any relevant operation.

Could you please give direction on how to include it if feasible?

Error Log:

java.util.NoSuchElementException: key not found: org.apache.spark.ml.feature.SQLTransformer at scala.collection.MapLike$class.default(MapLike.scala:228) at scala.collection.AbstractMap.default(Map.scala:59) at scala.collection.MapLike$class.apply(MapLike.scala:141) at scala.collection.AbstractMap.apply(Map.scala:59) at ml.combust.bundle.BundleRegistry.opForObj(BundleRegistry.scala:100) at ml.combust.bundle.serializer.GraphSerializer$$anonfun$writeNode$1.apply(GraphSerializer.scala:31) at ml.combust.bundle.serializer.GraphSerializer$$anonfun$writeNode$1.apply(GraphSerializer.scala:30) at scala.util.Try$.apply(Try.scala:192) at ml.combust.bundle.serializer.GraphSerializer.writeNode(GraphSerializer.scala:30) at ml.combust.bundle.serializer.GraphSerializer$$anonfun$write$2.apply(GraphSerializer.scala:21) at ml.combust.bundle.serializer.GraphSerializer$$anonfun$write$2.apply(GraphSerializer.scala:21) at scala.collection.IndexedSeqOptimized$class.foldl(IndexedSeqOptimized.scala:57) at scala.collection.IndexedSeqOptimized$class.foldLeft(IndexedSeqOptimized.scala:66) at scala.collection.mutable.WrappedArray.foldLeft(WrappedArray.scala:35) at ml.combust.bundle.serializer.GraphSerializer.write(GraphSerializer.scala:20) at org.apache.spark.ml.bundle.ops.PipelineOp$$anon$1.store(PipelineOp.scala:21) at org.apache.spark.ml.bundle.ops.PipelineOp$$anon$1.store(PipelineOp.scala:14) at ml.combust.bundle.serializer.ModelSerializer$$anonfun$write$1.apply(ModelSerializer.scala:92) at ml.combust.bundle.serializer.ModelSerializer$$anonfun$write$1.apply(ModelSerializer.scala:88) at scala.util.Try$.apply(Try.scala:192) at ml.combust.bundle.serializer.ModelSerializer.write(ModelSerializer.scala:88) at ml.combust.bundle.serializer.NodeSerializer$$anonfun$write$1.apply(NodeSerializer.scala:88) at ml.combust.bundle.serializer.NodeSerializer$$anonfun$write$1.apply(NodeSerializer.scala:84) at scala.util.Try$.apply(Try.scala:192) at ml.combust.bundle.serializer.NodeSerializer.write(NodeSerializer.scala:84) at ml.combust.bundle.serializer.GraphSerializer$$anonfun$writeNode$1.apply(GraphSerializer.scala:34) at ml.combust.bundle.serializer.GraphSerializer$$anonfun$writeNode$1.apply(GraphSerializer.scala:30) at scala.util.Try$.apply(Try.scala:192) at ml.combust.bundle.serializer.GraphSerializer.writeNode(GraphSerializer.scala:30) at ml.combust.bundle.serializer.GraphSerializer$$anonfun$write$2.apply(GraphSerializer.scala:21) at ml.combust.bundle.serializer.GraphSerializer$$anonfun$write$2.apply(GraphSerializer.scala:21) at scala.collection.IndexedSeqOptimized$class.foldl(IndexedSeqOptimized.scala:57) at scala.collection.IndexedSeqOptimized$class.foldLeft(IndexedSeqOptimized.scala:66) at scala.collection.mutable.WrappedArray.foldLeft(WrappedArray.scala:35) at ml.combust.bundle.serializer.GraphSerializer.write(GraphSerializer.scala:20) at org.apache.spark.ml.bundle.ops.PipelineOp$$anon$1.store(PipelineOp.scala:21) at org.apache.spark.ml.bundle.ops.PipelineOp$$anon$1.store(PipelineOp.scala:14) at ml.combust.bundle.serializer.ModelSerializer$$anonfun$write$1.apply(ModelSerializer.scala:92) at ml.combust.bundle.serializer.ModelSerializer$$anonfun$write$1.apply(ModelSerializer.scala:88) at scala.util.Try$.apply(Try.scala:192) at ml.combust.bundle.serializer.ModelSerializer.write(ModelSerializer.scala:88) at ml.combust.bundle.serializer.NodeSerializer$$anonfun$write$1.apply(NodeSerializer.scala:88) at ml.combust.bundle.serializer.NodeSerializer$$anonfun$write$1.apply(NodeSerializer.scala:84) at scala.util.Try$.apply(Try.scala:192) at ml.combust.bundle.serializer.NodeSerializer.write(NodeSerializer.scala:84) at ml.combust.bundle.serializer.BundleSerializer$$anonfun$write$1.apply(BundleSerializer.scala:34) at ml.combust.bundle.serializer.BundleSerializer$$anonfun$write$1.apply(BundleSerializer.scala:29) at scala.util.Try$.apply(Try.scala:192) at ml.combust.bundle.serializer.BundleSerializer.write(BundleSerializer.scala:29) at ml.combust.bundle.BundleWriter.save(BundleWriter.scala:27) at $anonfun$2.apply(:105) at $anonfun$2.apply(:104) at resource.AbstractManagedResource$$anonfun$5.apply(AbstractManagedResource.scala:88) at scala.util.control.Exception$Catch$$anonfun$either$1.apply(Exception.scala:125) at scala.util.control.Exception$Catch$$anonfun$either$1.apply(Exception.scala:125) at scala.util.control.Exception$Catch.apply(Exception.scala:103) at scala.util.control.Exception$Catch.either(Exception.scala:125) at resource.AbstractManagedResource.acquireFor(AbstractManagedResource.scala:88) at resource.ManagedResourceOperations$class.apply(ManagedResourceOperations.scala:26) at resource.AbstractManagedResource.apply(AbstractManagedResource.scala:50) at resource.ManagedResourceOperations$class.acquireAndGet(ManagedResourceOperations.scala:25) at resource.AbstractManagedResource.acquireAndGet(AbstractManagedResource.scala:50) at resource.ManagedResourceOperations$class.foreach(ManagedResourceOperations.scala:53) at resource.AbstractManagedResource.foreach(AbstractManagedResource.scala:50) ... 62 elided

Contributor guide

Tech stack
scala
Domain
machine learning
Issue type
feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-2 days
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
fresh
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
mostly clear
Prerequisites
ScalaSpark MLMLeap basics
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
30
Research direction
Study existing ops in mleap spark/src/main/scala/org/apache/spark/ml/bundle/ops/feature to understand the pattern. Implement a similar op for SQLTransformer, handling the UDF case. The PR and comments may provide additional context. Test with a pipeline containing SQLTransformer.