Integer features being converted to double in Mleap

Open
  • Is something wrong?
#737 2 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
35/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Stale
Tech stack
python, scala, spark
Domain
data-engineering, machine-learning

Research direction

Reproduce the issue using the shown PySpark save_model flow, the 0.17.0-SNAPSHOT serving container, and the model meta endpoint. Compare the integer and boolean input schema passed to serializeToBundle with the returned inputSchema, then trace the schema conversion path. Done means supported integer features are represented consistently rather than being reported as DOUBLE.

Written by the indexing model from the issue text.

Description

I've trained a Logistic Regression model with Pyspark, some of these are integers. When the model is loaded into a serving container the schema for the model has those features as floats. According to https://mleap-docs.combust.ml/core-concepts/data-frames/data-types.html Integers are supported

code for saving the model
def save_model(self, model, predictions):
    """
    Save model to a GCS bucket for use in inference
    """
    model_file = f"{self.model_name}.zip"
    tmp_location = f"/tmp/models/{model_file}"
    model.serializeToBundle(f"jar:file:{tmp_location}", predictions)
the schema of the predictions passed with the model

root
|-- client_id: string (nullable = true)
|-- dispatch_country: string (nullable = true)
|-- dw_classname: string (nullable = true)
|-- language_detected: string (nullable = true)
|-- photos_text_present: boolean (nullable = true)
|-- platform_id: string (nullable = true)
|-- site_group: string (nullable = true)
|-- territory: string (nullable = true)
|-- actioned: boolean (nullable = true)
|-- bcp_brands__count: integer (nullable = true)
|-- bcp_collections__count: integer (nullable = true)
|-- bcp_products__count: integer (nullable = true)
|-- photos_matched_logo_types__count: integer (nullable = true)
|-- image_match_types__count: integer (nullable = true)
|-- photos_matched_friendly_id__count: integer (nullable = true)

The code for loading the saved model

docker pull combustml/mleap-spring-boot:0.17.0-SNAPSHOT
docker run -p 8080:8080 -v /tmp/models:/models -e SERVER_PORT=8080 combustml/mleap-spring-boot:0.17.0-SNAPSHOT

body='{"modelName":"enforcement_structured","uri":"file:/models/enforcement_structured.zip","config":{"memoryTimeout":900000,"diskTimeout":900000},"force":false}'
curl --header "Content-Type: application/json" --request POST --data "$body" http://localhost:8080/models

The schema of calling the meta endpoint

curl --header "Content-Type: application/json" --request GET http://localhost:8080/models/enforcement_structured/meta

{"bundle":{"uid":"b837d936-7d86-4c94-86ad-d215b7285297","name":"PipelineModel_aa1fc762aa64","format":"JSON","version":"0.16.0","timestamp":"2021-01-27T12:42:19.363"},

"inputSchema":{"fields":[
{"name":"site_group","dataType":{"base":"STRING","shape":{"base":"SCALAR","isNullable":true}}},

{"name":"language_detected","dataType":{"base":"STRING","shape":{"base":"SCALAR","isNullable":true}}},

{"name":"image_match_types__count","dataType":{"base":"DOUBLE","shape":{"base":"SCALAR","isNullable":true}}},

{"name":"client_id","dataType":{"base":"STRING","shape":{"base":"SCALAR","isNullable":true}}},

{"name":"bcp_brands__count","dataType":{"base":"DOUBLE","shape":{"base":"SCALAR","isNullable":true}}},

{"name":"photos_text_present","dataType":{"base":"DOUBLE","shape":{"base":"SCALAR","isNullable":true}}},

{"name":"dispatch_country","dataType":{"base":"STRING","shape":{"base":"SCALAR","isNullable":true}}},

{"name":"bcp_products__count","dataType":{"base":"DOUBLE","shape":{"base":"SCALAR","isNullable":true}}},

{"name":"photos_matched_friendly_id__count","dataType":{"base":"DOUBLE","shape":{"base":"SCALAR","isNullable":true}}},

{"name":"territory","dataType":{"base":"STRING","shape":{"base":"SCALAR","isNullable":true}}},

{"name":"bcp_collections__count","dataType":{"base":"DOUBLE","shape":{"base":"SCALAR","isNullable":true}}},

{"name":"dw_classname","dataType":{"base":"STRING","shape":{"base":"SCALAR","isNullable":true}}},

{"name":"photos_matched_logo_types__count","dataType":{"base":"DOUBLE","shape":{"base":"SCALAR","isNullable":true}}},

{"name":"platform_id","dataType":{"base":"STRING","shape":{"base":"SCALAR","isNullable":true}}}]},

"outputSchema":{"fields":[{"name":"platform_id_indexed","dataType":{"base":"DOUBLE","shape":{"base":"SCALAR","isNullable":false}}},{"name":"features","dataType":{"base":"DOUBLE","shape":{"base":"TENSOR","isNullable":true,"tensorShape":{"dimensions":[{"size":14,"name":""}]}}}},{"name":"territory_indexed","dataType":{"base":"DOUBLE","shape":{"base":"SCALAR","isNullable":false}}},{"name":"prediction","dataType":{"base":"DOUBLE","shape":{"base":"SCALAR","isNullable":false}}},{"name":"client_id_indexed","dataType":{"base":"DOUBLE","shape":{"base":"SCALAR","isNullable":false}}},{"name":"language_detected_indexed","dataType":{"base":"DOUBLE","shape":{"base":"SCALAR","isNullable":false}}},{"name":"site_group_indexed","dataType":{"base":"DOUBLE","shape":{"base":"SCALAR","isNullable":false}}},{"name":"dispatch_country_indexed","dataType":{"base":"DOUBLE","shape":{"base":"SCALAR","isNullable":false}}},{"name":"dw_classname_indexed","dataType":{"base":"DOUBLE","shape":{"base":"SCALAR","isNullable":false}}},{"name":"rawPrediction","dataType":{"base":"DOUBLE","shape":{"base":"TENSOR","isNullable":true,"tensorShape":{"dimensions":[{"size":2,"name":""}]}}}},{"name":"probability","dataType":{"base":"DOUBLE","shape":{"base":"TENSOR","isNullable":true,"tensorShape":{"dimensions":[{"size":2,"name":""}]}}}}]}}
Dominant language
Scala
Stars
1.5k
Forks
313
Avg merge
22m
Merged PRs (30d)
1

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

  • No Dockerfile or Docker Compose file
  • No pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from combust/mleap

All issues in combust/mleap

Similar issues

More Scala issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.