Implement Spark-compatible `schema_of_variant`

Aperta
  • Qualcosa non va?
#278 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
45/100
Tipo di issue
Funzionalità
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
rust, sql
Ambito
backend, data

Direzione di ricerca

Read Spark's SchemaOfVariant in variantExpressions.scala alongside JsonInferSchema.compatibleType and TypeCoercion.findTightestCommonType to get the merge lattice, then look at how existing functions under crates/spark/src/function/variant/ walk parquet_variant::Variant values. Implement schemaOf, mergeSchema and printSchema reusing that walking pattern, and register the function with #[user_doc]. Done means crates/sqllogictest/test_files/variant/schema_of_variant.slt reproduces the example table (quoted field names, OBJECT<...>, decimal stripping) against Spark output.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

enhancement

Part of #273 and apache/datafusion-comet#7300.

What the function does

schema_of_variant(v) returns a STRING describing the schema of a single VARIANT value. The format is Spark's
SQL type syntax, except that objects print as OBJECT<...> instead of STRUCT<...>. Variant objects are unordered
bags of fields, unlike structs. The argument must be VARIANT and the result uses the default string type (and
collation). SQL NULL input returns NULL through StaticInvoke null propagation. Added in Spark 4.0.0.

Type mapping (SchemaOfVariant.schemaOf):

Variant type Result
null VOID
boolean BOOLEAN
INT1, INT2, INT4, INT8 BIGINT
string STRING
double DOUBLE
float FLOAT
decimal DECIMAL(p,s), from the value after trailing zeros are stripped
date, timestamp, timestamp_ntz DATE, TIMESTAMP, TIMESTAMP_NTZ
binary BINARY
uuid UUID
object OBJECT<name: type, ...> in key order
array ARRAY<merged element type>

Array elements are merged with JsonInferSchema.compatibleType(t1, t2, VariantType), so incompatible types become
VARIANT.

Examples
SELECT schema_of_variant(parse_json('null'));
-- VOID
SELECT schema_of_variant(parse_json('[{"b":true,"a":0}]'));
-- ARRAY<OBJECT<a: BIGINT, b: BOOLEAN>>
SELECT schema_of_variant(parse_json('1.0')), schema_of_variant(parse_json('10.10')), schema_of_variant(parse_json('-0.0'));
-- DECIMAL(1,0)  DECIMAL(3,1)  DECIMAL(1,0)
SELECT schema_of_variant(parse_json('9223372036854775808'));
-- DECIMAL(19,0)
SELECT schema_of_variant(parse_json('9999999999999999999.99999999999999999999'));
-- DOUBLE
SELECT schema_of_variant(parse_json('"2000-01-01"'));
-- STRING
SELECT schema_of_variant(parse_json('[]'));
-- ARRAY<VOID>
SELECT schema_of_variant(parse_json('[null, 1, 1.1]'));
-- ARRAY<DECIMAL(21,1)>
SELECT schema_of_variant(parse_json('[1.1, 11111111111111111111111111111111111111]'));
-- ARRAY<DOUBLE>
SELECT schema_of_variant(parse_json('[1, "1"]'));
-- ARRAY<VARIANT>
SELECT schema_of_variant(parse_json('[{"c": ""}, {"a": null}, {"b": 1}]'));
-- ARRAY<OBJECT<a: VOID, b: BIGINT, c: STRING>>
SELECT schema_of_variant(parse_json('[{"a": 1, "b": null}, {"b": true, "a": 1E0}]'));
-- ARRAY<OBJECT<a: DOUBLE, b: BOOLEAN>>
SELECT schema_of_variant(parse_json('{"STRUCT": {"!special!": true}}'));
-- OBJECT<STRUCT: OBJECT<`!special!`: BOOLEAN>>

These come from the ExpressionDescription, VariantEndToEndSuite and VariantSuite.

Edge cases to test

  • Merge table from VariantExpressionSuite."schema_of_variant - schema merge":

    Pair Result
    BIGINT/DOUBLE DOUBLE
    BIGINT/FLOAT FLOAT
    BIGINT/DECIMAL(6,3) DECIMAL(23,3)
    DOUBLE/FLOAT DOUBLE
    DOUBLE/DECIMAL DOUBLE
    FLOAT/DECIMAL DOUBLE
    DATE/TIMESTAMP TIMESTAMP
    DATE/TIMESTAMP_NTZ TIMESTAMP_NTZ
    TIMESTAMP/TIMESTAMP_NTZ TIMESTAMP
    ARRAY<BIGINT>/ARRAY<DOUBLE> ARRAY<DOUBLE>
    OBJECT<a: STRING>/OBJECT<a: BOOLEAN, b: BIGINT> OBJECT<a: VARIANT, b: BIGINT>

    VOID merged with anything gives the other type, and every other pair gives VARIANT, including
    STRING/BIGINT and BINARY/STRING.

  • Decimal merging: the scale is max(s1, s2) and the integer range is max(p - s). When the total exceeds 38 the
    result is DOUBLE. An integral type is widened to DECIMAL(20,0) first ([null, 1, 1.0] gives
    DECIMAL(20,0), and [123456.789, 123.456789] gives DECIMAL(12,6)).

  • Decimal precision comes from stripTrailingZeros, with a negative scale normalized to 0: 1.00 gives
    DECIMAL(1,0) and 10.00 gives DECIMAL(2,0). It does not come from the stored scale.

  • Field names are quoted with backticks unless they match [a-zA-Z_][a-zA-Z0-9_]*, and embedded backticks are
    doubled (OBJECT<`it's`: ...>). Keywords like STRUCT are not quoted.

  • Struct merging inside arrays first goes through TypeCoercion.findTightestCommonType. For structs of equal
    length, that compares names with spark.sql.caseSensitive (default false) and keeps the first name. Only if it
    fails does Spark fall back to a case-sensitive sorted merge. By code reading, [{"a":1},{"A":2}] gives
    ARRAY<OBJECT<a: BIGINT>> by default. Verify this before asserting it.

  • Object fields must be strictly increasing by Java String.compareTo (UTF-16 order). Otherwise Spark throws
    MALFORMED_VARIANT, and duplicates hit the same check. Variants written by other tools that sort by UTF-8 bytes
    can trip this.

  • An unknown primitive type id raises UNKNOWN_PRIMITIVE_TYPE_IN_VARIANT (id 25 in the suite).

  • UUID shows as UUID, and merging it with a variant null keeps UUID (ARRAY<UUID>). These values can only come
    from external data.

  • Test NULL input, column input, and deep nesting.

Differences across Spark versions
  • 3.x has no VARIANT.
  • No logic changes between 4.0.4, 4.1.3 and 4.2.0. 4.2 adds GEOMETRY/GEOGRAPHY cases to
    findTightestCommonType, but variants never produce those types.
Implementation notes
  • Spark: SchemaOfVariant (schemaOf, mergeSchema, printSchema) in
    sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/variantExpressions.scala.
    The merge rules are JsonInferSchema.compatibleType in .../catalyst/json/JsonInferSchema.scala and
    TypeCoercion.findTightestCommonType / TypeCoercionHelper.findTypeForComplex in .../catalyst/analysis/.
    The printer is DataType.sql, with QuotingUtils.quoteIfNeeded for field names.
  • DataFusion core and arrow-rs have nothing equivalent. Reuse parquet_variant::Variant to walk values and write
    the Spark type lattice and printer directly. Share them with schema_of_variant_agg, and with
    schema_of_json / from_json's merge logic if they land.
  • This depends on the VARIANT Arrow representation in the parse_json issue.
  • apache/datafusion-comet#5438 tracks native Variant support in Comet.
  • Put it in crates/spark/src/function/variant/. Add crates/sqllogictest/test_files/variant/schema_of_variant.slt
    and document the function with #[user_doc].

This issue was generated by an LLM (Claude) from the Apache Spark source code and tests. Examples come from Spark's ExpressionDescription examples, test suites and .sql.out golden files, but please verify them against Spark before relying on them.

Lingua principale
Rust
Stelle
7
Fork
15
Merge medio
16h 43m
PR unite (30g)
26

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di apache/datafusion-spark

Tutte le issue di apache/datafusion-spark

Issue simili

Altre issue su Rust

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.