Impossible to deserialize a bundle written with Scikit-Learn with Pyspark: no "bundle.json found"

Open
  • Is something wrong?
#691 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
35/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Stale
Tech stack
python, scikit-learn
Domain
machine-learning

Research direction

Start with the scikit-learn serialize_to_bundle call and the PySpark RandomForestClassificationModel.deserializeFromBundle entry point shown in the reproduction. Trace the expected bundle layout and determine whether the generated model can be consumed by the Spark reader; done means the compatibility behavior is verified and the missing bundle.json failure is resolved or clearly documented.

Written by the indexing model from the issue text.

Description

I serialize a model with Scikit-Learn:

#Generate data
import pandas as pd 
import numpy as np

df = pd.DataFrame(np.random.randn(100, 5), columns=['a', 'b', 'c', 'd', 'e'])
df["y"] = (df['a'] > 0.5).astype(int)
df.head()

from mleap.sklearn.ensemble.forest import RandomForestClassifier

forestModel = RandomForestClassifier()
forestModel.mlinit(input_features='a',
                   feature_names='a',
                           prediction_column='e_binary')


forestModel.fit(df[['a']], df[['y']])

forestModel.serialize_to_bundle("/dbfs/FileStore/tables/mleaptestmodelforest", "model.json")

When I try to read it with Pyspark:

from pyspark.ml.classification import RandomForestClassificationModel

model = RandomForestClassificationModel.deserializeFromBundle("file:/dbfs/FileStore/tables/mleaptestmodelforest")

I have this error:
java.nio.file.NoSuchFileException: /dbfs/FileStore/tables/mleaptestmodelforest/bundle.json

I have no "bundle.json".

Could you help me please?
Is it really possible to seralize a model with Scikit-Learn and deserialize it with Pyspark?

Dominant language
Scala
Stars
1.5k
Forks
313
Avg merge
22m
Merged PRs (30d)
1

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

  • No Dockerfile or Docker Compose file
  • No pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from combust/mleap

All issues in combust/mleap

Similar issues

More Scala issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.