Implement Spark-compatible `schema_of_variant`

Abierto
  • ¿Algo no está bien?
#278 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
45/100
Tipo de issue
Nueva funcionalidad
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
rust, sql
Área
backend, data

Línea de trabajo

Read Spark's SchemaOfVariant in variantExpressions.scala alongside JsonInferSchema.compatibleType and TypeCoercion.findTightestCommonType to get the merge lattice, then look at how existing functions under crates/spark/src/function/variant/ walk parquet_variant::Variant values. Implement schemaOf, mergeSchema and printSchema reusing that walking pattern, and register the function with #[user_doc]. Done means crates/sqllogictest/test_files/variant/schema_of_variant.slt reproduces the example table (quoted field names, OBJECT<...>, decimal stripping) against Spark output.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

enhancement

Part of #273 and apache/datafusion-comet#7300.

What the function does

schema_of_variant(v) returns a STRING describing the schema of a single VARIANT value. The format is Spark's
SQL type syntax, except that objects print as OBJECT<...> instead of STRUCT<...>. Variant objects are unordered
bags of fields, unlike structs. The argument must be VARIANT and the result uses the default string type (and
collation). SQL NULL input returns NULL through StaticInvoke null propagation. Added in Spark 4.0.0.

Type mapping (SchemaOfVariant.schemaOf):

Variant type Result
null VOID
boolean BOOLEAN
INT1, INT2, INT4, INT8 BIGINT
string STRING
double DOUBLE
float FLOAT
decimal DECIMAL(p,s), from the value after trailing zeros are stripped
date, timestamp, timestamp_ntz DATE, TIMESTAMP, TIMESTAMP_NTZ
binary BINARY
uuid UUID
object OBJECT<name: type, ...> in key order
array ARRAY<merged element type>

Array elements are merged with JsonInferSchema.compatibleType(t1, t2, VariantType), so incompatible types become
VARIANT.

Examples
SELECT schema_of_variant(parse_json('null'));
-- VOID
SELECT schema_of_variant(parse_json('[{"b":true,"a":0}]'));
-- ARRAY<OBJECT<a: BIGINT, b: BOOLEAN>>
SELECT schema_of_variant(parse_json('1.0')), schema_of_variant(parse_json('10.10')), schema_of_variant(parse_json('-0.0'));
-- DECIMAL(1,0)  DECIMAL(3,1)  DECIMAL(1,0)
SELECT schema_of_variant(parse_json('9223372036854775808'));
-- DECIMAL(19,0)
SELECT schema_of_variant(parse_json('9999999999999999999.99999999999999999999'));
-- DOUBLE
SELECT schema_of_variant(parse_json('"2000-01-01"'));
-- STRING
SELECT schema_of_variant(parse_json('[]'));
-- ARRAY<VOID>
SELECT schema_of_variant(parse_json('[null, 1, 1.1]'));
-- ARRAY<DECIMAL(21,1)>
SELECT schema_of_variant(parse_json('[1.1, 11111111111111111111111111111111111111]'));
-- ARRAY<DOUBLE>
SELECT schema_of_variant(parse_json('[1, "1"]'));
-- ARRAY<VARIANT>
SELECT schema_of_variant(parse_json('[{"c": ""}, {"a": null}, {"b": 1}]'));
-- ARRAY<OBJECT<a: VOID, b: BIGINT, c: STRING>>
SELECT schema_of_variant(parse_json('[{"a": 1, "b": null}, {"b": true, "a": 1E0}]'));
-- ARRAY<OBJECT<a: DOUBLE, b: BOOLEAN>>
SELECT schema_of_variant(parse_json('{"STRUCT": {"!special!": true}}'));
-- OBJECT<STRUCT: OBJECT<`!special!`: BOOLEAN>>

These come from the ExpressionDescription, VariantEndToEndSuite and VariantSuite.

Edge cases to test

  • Merge table from VariantExpressionSuite."schema_of_variant - schema merge":

    Pair Result
    BIGINT/DOUBLE DOUBLE
    BIGINT/FLOAT FLOAT
    BIGINT/DECIMAL(6,3) DECIMAL(23,3)
    DOUBLE/FLOAT DOUBLE
    DOUBLE/DECIMAL DOUBLE
    FLOAT/DECIMAL DOUBLE
    DATE/TIMESTAMP TIMESTAMP
    DATE/TIMESTAMP_NTZ TIMESTAMP_NTZ
    TIMESTAMP/TIMESTAMP_NTZ TIMESTAMP
    ARRAY<BIGINT>/ARRAY<DOUBLE> ARRAY<DOUBLE>
    OBJECT<a: STRING>/OBJECT<a: BOOLEAN, b: BIGINT> OBJECT<a: VARIANT, b: BIGINT>

    VOID merged with anything gives the other type, and every other pair gives VARIANT, including
    STRING/BIGINT and BINARY/STRING.

  • Decimal merging: the scale is max(s1, s2) and the integer range is max(p - s). When the total exceeds 38 the
    result is DOUBLE. An integral type is widened to DECIMAL(20,0) first ([null, 1, 1.0] gives
    DECIMAL(20,0), and [123456.789, 123.456789] gives DECIMAL(12,6)).

  • Decimal precision comes from stripTrailingZeros, with a negative scale normalized to 0: 1.00 gives
    DECIMAL(1,0) and 10.00 gives DECIMAL(2,0). It does not come from the stored scale.

  • Field names are quoted with backticks unless they match [a-zA-Z_][a-zA-Z0-9_]*, and embedded backticks are
    doubled (OBJECT<`it's`: ...>). Keywords like STRUCT are not quoted.

  • Struct merging inside arrays first goes through TypeCoercion.findTightestCommonType. For structs of equal
    length, that compares names with spark.sql.caseSensitive (default false) and keeps the first name. Only if it
    fails does Spark fall back to a case-sensitive sorted merge. By code reading, [{"a":1},{"A":2}] gives
    ARRAY<OBJECT<a: BIGINT>> by default. Verify this before asserting it.

  • Object fields must be strictly increasing by Java String.compareTo (UTF-16 order). Otherwise Spark throws
    MALFORMED_VARIANT, and duplicates hit the same check. Variants written by other tools that sort by UTF-8 bytes
    can trip this.

  • An unknown primitive type id raises UNKNOWN_PRIMITIVE_TYPE_IN_VARIANT (id 25 in the suite).

  • UUID shows as UUID, and merging it with a variant null keeps UUID (ARRAY<UUID>). These values can only come
    from external data.

  • Test NULL input, column input, and deep nesting.

Differences across Spark versions
  • 3.x has no VARIANT.
  • No logic changes between 4.0.4, 4.1.3 and 4.2.0. 4.2 adds GEOMETRY/GEOGRAPHY cases to
    findTightestCommonType, but variants never produce those types.
Implementation notes
  • Spark: SchemaOfVariant (schemaOf, mergeSchema, printSchema) in
    sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/variantExpressions.scala.
    The merge rules are JsonInferSchema.compatibleType in .../catalyst/json/JsonInferSchema.scala and
    TypeCoercion.findTightestCommonType / TypeCoercionHelper.findTypeForComplex in .../catalyst/analysis/.
    The printer is DataType.sql, with QuotingUtils.quoteIfNeeded for field names.
  • DataFusion core and arrow-rs have nothing equivalent. Reuse parquet_variant::Variant to walk values and write
    the Spark type lattice and printer directly. Share them with schema_of_variant_agg, and with
    schema_of_json / from_json's merge logic if they land.
  • This depends on the VARIANT Arrow representation in the parse_json issue.
  • apache/datafusion-comet#5438 tracks native Variant support in Comet.
  • Put it in crates/spark/src/function/variant/. Add crates/sqllogictest/test_files/variant/schema_of_variant.slt
    and document the function with #[user_doc].

This issue was generated by an LLM (Claude) from the Apache Spark source code and tests. Examples come from Spark's ExpressionDescription examples, test suites and .sql.out golden files, but please verify them against Spark before relying on them.

Lenguaje dominante
Rust
Estrellas
7
Forks
15
Merge medio
16 h 43 min
PR fusionados (30 d)
26

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de apache/datafusion-spark

Todos los issues de apache/datafusion-spark

Issues similares

Más issues de Rust

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.