[RFC] Widen numeric mapping conflicts across indices to one deterministic type

Read MergeRuleHelper.java and LatestRule.java to see how mapping conflicts are currently resolved, then review the proposed widening table and WideningTypeRule.java. The implementation discussion estimates about 80 lines for NumericWideningRule, but linked PR #5902 is already open. Done means numeric conflicts merge deterministically across index read orders while respecting the stated exclusions and behavior.

Problem Statement

When the indices behind one source such as source=logs-* map the same field to two different number types, the plugin keeps the type of whichever index the merge reads last and decodes every document under it. Which index wins depends on the index names and can change between nodes and after a restart. Under the wrong winner a value changes or disappears with HTTP 200. A long 5000000000 read as integer overflows and comes back as 705032704, and a double 1.5 read as bigint comes back as null. A rollover that changes one field from integer to long is enough to trigger it.

#5828 reports the changed and null values, #5811 the 500 from a group-by on such a field, and #5610 asks for a conflict policy. A number against a string type, such as keyword against long, and conflicts that involve date or other special types will be proposed in separate RFCs.

Current State

MergeRuleHelper sends every conflict between two number types to LatestRule, and since #5618 a value that fails to decode under the winning type becomes null.

A type sweep of main at 00262797c ran all 28 pairs of the eight number types, byte, short, integer, long, half_float, float, scaled_float and double, in both read orders, through about 100 PPL templates. In 26 of the 28 pairs the merged type depends on the index names, 16 return a null in fields, and 16 return a 500 that neither type returns alone.

Scope

This RFC covers a field mapped as two or more number types across the indices of one source.

• Every engine - PPL on Calcite, PPL on the V2 engine and SQL share one mapping merge, so all three get the merged type.
• Number types only - The guarantee holds when every index maps the field as a number.

Out of scope:

1. A number in some indices and a string in others - Mixed datatypes such as long, keyword and integer will be a follow-up on a number against a string type.
2. alias - An alias that points at a number field has no mapping type of its own, so it still reaches LatestRule, and when it meets a concrete number field the losing index's values can read as null.
3. unsigned_long - The plugin does not support the type, so the merge never sees it. Support was requested in #1413 and closed as not planned.
4. Untested routes - Cross-cluster sources and the analytics engine route have not been tested.

Long-Term Goals

• A field mapped as different number types gets one merged type, whatever order the indices are read in.
• The merged type holds every value of every index, apart from a long beyond 2^53 next to a floating point type. A pushed float aggregate may print more digits of the same stored value.
• Every command either runs on the merged type or returns the same 4xx it returns on a single field of that type. A number conflict never causes a 5xx.

Proposal

A new NumericWideningRule sits ahead of LatestRule and merges number types to the widest one. The merged type depends only on which number types the indices hold, whatever the number of indices.

Number types across the indices	Merged type
integer types only, from byte, short, integer and long	the widest of them
half_float and float only	float
floating point types only, with scaled_float or double among them	double
at least one integer type and at least one floating point type	double

The ordering follows core's WideningTypeRule with one change. An integer with a floating point type merges to double, where WideningTypeRule picks float. A float holds integers exactly only up to 2^24, so 16777217 would read 16777216.0. The rule keeps its own copy, because V2 uses WideningTypeRule to pick function signatures.

The merged type decides how arithmetic behaves.

• An integer with a floating point type gives double, so f / 2 on 3 gives 1.5. main already does this when the floating point index wins.
• Integer types only give an integer type, so f / 2 on 3 still gives 1 under the default plugins.ppl.syntax.legacy.preferred.
• floor(f / 2) gives integer division on a double field.

Integer with float, measured under each merged type

An integer index holding 16777217 and a float index holding 0.1, under three sets of index names each.

Merged type	fields f	stats min(f), max(f)
float, on main when float wins	0.1, 16777216.0	0.1, 16777216.0
int, on main when integer wins	null, 16777217	0, 16777217
double, with the rule	0.1, 16777217.0	0.10000000149011612, 16777217.0

Approach

Pushdown with a merged type

The merged type decides how the plugin decodes values and plans the query, while each shard reads a pushed field under its own mapping, so a byte shard returns Integer, a long shard Long and a double shard Double. A pushed operation whose reduce compares those values across shards can fail whatever the merged type is. Raw _search requests show which ones.

Raw _search per pair, the composite terms and top_hits sort fail when the shards return different Java types

Pair	plain sort	terms	composite terms	max	top_hits sort
integer with long	200	200	200	200	500, passes with numeric_type: long
integer with double	200	200	500	200	500, passes with numeric_type: double
long with double	200	200	500	200	500, passes with numeric_type: double
float with double	200	200	200	200	500, passes with numeric_type: double

Two kinds of PPL query reach the failing cells, and both already fail on main without the rule.

• #5811
• #5897

Ten other commands, among them fields, sort, where, stats and top, matched pushdown off on four pairs, on a source of all eight number types and on a source of the four integer types.

Costs

• An integer row next to a floating point type renders as a double, so 3 reads 3.0. This is the most common case, since dynamic mapping maps a JSON integer to long and a JSON decimal to float.
• A pushed stats min over a float index prints 0.1 as 0.10000000149011612, the stored float's exact value, when another index maps the field as a wider number.
• A long beyond 2^53 next to a floating point type rounds when read, in a group key and in a distinct count. A pushed where f = 9007199254740993 still matches only that document.
• half_float with float and scaled_float with double change nothing, since PPL already reads them as float and double.

Alternative

• Merge every number conflict to double. One rule with no ordering. It turns integer-only conflicts such as integer with long into double too, so those rows read 3.0 and a long above 2^53 rounds, where widening to long loses nothing.
• Widen an integer with float to float, as core's WideningTypeRule does. It keeps the float name, and it changes every integer above 2^24, so 16777217 reads 16777216.0.
• Merge number conflicts to string. Every value keeps its stored text exactly, but the field stops being a number, so sort, max and min order the values as text and percentile rejects them, even though a number type can hold every value.
• Decode each document with its own index's mapping. It fixes the decode without choosing a winner, and it still needs a merged type for the schema and for pushdown. It changes the hit path, the aggregation parsers and the top_hits parser.

Implementation Discussion

• The policy is one PR, NumericWideningRule, about 80 lines of main code.
• Two existing bugs become simpler to fix once it lands, and each follows as its own fix.
  • #5811 can read a field merged from integer and floating point types through a script that returns a double, since the merged type is then always double, where on main it may be an integer and the script would truncate the floating point values.
  • #5897 can also set numeric_type on the dedup sort from the merged type