elastic/elasticsearch

ignore_malformed to support ignoring JSON objects ingested into fields of the wrong type

Open

#12366 opened on Jul 21, 2015

View on GitHub
 (73 comments) (68 reactions) (0 assignees)Java (76,700 stars) (25,882 forks)batch import
:StorageEngine/Mapping>enhancementTeam:StorageEnginehelp wanted

Description

Indexing a document with an object type on a field that has already been mapped as a string type causes MapperParsingException, even if index.mapping.ignore_malformed has been enabled.

Reproducible test case

On Elasticsearch 1.6.0:

$ curl -XPUT localhost:9200/broken -d'{"settings":{"index.mapping.ignore_malformed": true}}'
{"acknowledged":true}

$ curl -XPOST localhost:9200/broken/type -d '{"test":"a string"}'
{"_index":"broken","_type":"type","_id":"AU6wNDGa_qDGqxty2Dvw","_version":1,"created":true}

$ curl -XPOST localhost:9200/broken/type -d '{"test":{"nested":"a string"}}'
{"error":"MapperParsingException[failed to parse [test]]; nested: ElasticsearchIllegalArgumentException[unknown property [nested]]; ","status":400}

$ curl localhost:9200/broken/_mapping
{"broken":{"mappings":{"type":{"properties":{"test":{"type":"string"}}}}}}

Expected behaviour

Indexing a document with an object field where Elasticsearch expected a string field to be will not fail the whole document when index.mapping.ignore_malformed is enabled. Instead, it will ignore the invalid object field.

Contributor guide

Tech stack
javarest api
Domain
databasebackendapi
Issue type
feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
4
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
3-5 days
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
clear
Prerequisites
JavaElasticsearch mapping conceptsMapperParsingException handling
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
25
Research direction
Review the discussion in issue #12366 to understand previous attempts and suggestions. Look at the document parsing code in Elasticsearch, particularly the DocumentParser class and how ignore malformed is handled for scalar fields. Extend that logic to object fields when they conflict with existing string mapping. Pay attention to the MapperParsingException flow and ensure the malformed object is silently dropped when ignore malformed is true.