combust/mleap

Integrate with spark-nlp

Open

#292 opened on Oct 21, 2017

View on GitHub
 (1 comment) (0 reactions) (0 assignees)Scala (1,461 stars) (312 forks)batch import
enhancementepichelp wanted

Description

spark-nlp

Is a library that was developed by john snow labs to offer common transformers for NLP to Spark. It would be incredibly useful to be able to train these models and then deploy them with MLeap.

https://github.com/JohnSnowLabs/spark-nlp

We still need to outline the work in terms of which transformers and models we need to offer support for and if we will include them in core modules or create separate modules for NLP.

Contributor guide

Tech stack
scalaspark
Domain
datamachine learning
Issue type
feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
5
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
over 1 week
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
needs investigation
Prerequisites
ScalaApache SparkNLP basicsMLeap internals
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
15
Research direction
Investigate the current MLeap module structure in the combust/mleap repository, particularly how existing transformers are defined and serialized. Review the spark nlp library by John Snow Labs (github.com/JohnSnowLabs/spark nlp) to identify which transformers and models are most commonly used. Then propose a plan to either extend core MLeap modules or create separate NLP specific modules, referencing specific files such as MleapTransformer.scala or similar. The discussion in the issue indicates that the scope needs to be defined first.