Implement Spark-compatible `variant_get` / `try_variant_get`

Start in crates/spark/src/function/variant/, reading the existing VARIANT functions and confirming the Arrow VARIANT representation from #273 has landed, since both functions depend on it. Then port Spark's VariantPathParser and VariantGet.cast dispatch, decoding values through parquet_variant::Variant, and register the UDF with return_field_from_args. Done means both names registered and documented with #[user_doc], plus crates/sqllogictest/test_files/variant/variant_get.slt covering the path grammar and cast edge cases listed above.

Part of #273 and apache/datafusion-comet#7300.

What the function does

variant_get(v, path[, type]) takes the sub-variant of v at path and casts it to type. try_variant_get
does the same, but a failed cast returns NULL instead of raising an error.

• v must be VARIANT. path is a STRING (any collation) and may be a column.
• type must be a foldable string. It is parsed as DDL or JSON schema, so 'int', 'array<int>' and
  'struct<a int>' all work. CHAR/VARCHAR are rejected.
• Without type the result is VARIANT. The result is always nullable, whatever nullability type declares.
• A NULL v or path returns NULL.
• Allowed target types are numerics, boolean, string, binary, date/timestamp/timestamp_ntz, VARIANT, arrays of
  these, MAP<STRING, ...> and structs. Anything else fails analysis with
  DATATYPE_MISMATCH.CAST_WITHOUT_SUGGESTION. That includes intervals and maps with non-string keys.
• Both functions were added in Spark 4.0.0. They depend on the VARIANT Arrow representation proposed in the
  parse_json issue.

The cast code (VariantGet.cast) is shared with CAST(v AS type) / TRY_CAST and the v:path extraction
syntax, so implementing it here covers those too.

Examples

SELECT variant_get(parse_json('{"a": 1}'), '$.a', 'int');
-- 1
SELECT variant_get(parse_json('{"a": 1}'), '$.b', 'int');
-- NULL
SELECT variant_get(parse_json('[1, "2"]'), '$[1]', 'string');
-- 2
SELECT variant_get(parse_json('[1, "2"]'), '$[2]', 'string');
-- NULL
SELECT variant_get(parse_json('[1, "hello"]'), '$[1]');
-- "hello"
SELECT try_variant_get(parse_json('[1, "hello"]'), '$[1]', 'int');
-- NULL
SELECT variant_get(parse_json('{"a": 2147483648}'), '$.a', 'int');
-- error INVALID_VARIANT_CAST: The variant value `2147483648` cannot be cast into `"INT"`. ...
SELECT variant_get(parse_json('{"1": {"2": {"3": [4]}}}'), '$.1.2.3[0]', 'int');
-- 4
SELECT variant_get(parse_json('{"1": {"2": {"3": [4]}}}'), '$.1.2.3[''0'']', 'int');
-- NULL
SELECT variant_get(parse_json('1.10'), '$', 'string');
-- 1.1
SELECT try_variant_get(parse_json('[1, 2, 3, "hello"]'), '$', 'array<int>');
-- [1,2,3,null]
SELECT variant_get(parse_json('{"a": 1, "d": 2, "c": 3}'), '$', 'struct<a int, b int, c int>');
-- {"a":1,"b":null,"c":3}

These come from the ExpressionDescription and VariantExpressionSuite. The struct and array results there are
asserted as rows, so the display form shown is the usual Spark rendering.

Edge cases to test

• Path grammar (VariantPathParser):
  • A path is $ followed by any number of .key, ['key'], ["key"] or [N] segments, with no whitespace
    skipping.
  • .key takes any characters except . and [, so $.zip code and $.fb:testid are valid.
  • Quoted keys cannot contain their own quote character, and there are no escapes.
  • An empty key needs [''] or [""].
  • Invalid paths raise INVALID_VARIANT_GET_PATH with functionName = `variant_get`, even in
    try_variant_get. Examples are "", .a, $1, $[-1] and $['"]. For a literal path the error is raised
    at first evaluation, not at analysis.
  • $[99999999999] matches \d+ and then toInt overflows. Pin down what Spark does there.
• Lookups:
  • Key lookup is case-sensitive ($.A does not match a).
  • A key segment on a non-object returns NULL, and so does an index segment on a non-array or out of range.
    Cover objects with more than 32 fields (binary search) and more than 255 elements or fields.
• JSON null versus SQL NULL:
  • A variant null cast to any concrete type, including STRING, is SQL NULL.
  • With no type (VARIANT), the variant null is kept, so is_variant_null(variant_get(..., '$.a')) is true.
• A VARIANT result is re-encoded so it only carries the dictionary keys it uses. It is not a byte slice of the
  input.
• Scalar casts:
  • A typed literal is built for the variant's type, then cast with Spark Cast in TRY mode, but only when
    Cast.canAnsiCast allows the pair. A NULL result or a disallowed pair counts as an invalid cast.
  • spark.sql.ansi.enabled has no effect. Both settings throw for variant_get.
• Spark test expectations to copy:
  • "true" to boolean is true; " t " to boolean is true.
  • "true" to int is an error.
  • true to decimal(18,17) is 1, and to decimal(18,18) is an error.
  • 128 to tinyint is an error.
  • 1.15 to decimal(2,1) is 1.2 (HALF_UP).
  • 2147483647.999 to int is 2147483647 (truncation).
  • "inf" and "nan" to double.
  • 1E308 to float is Infinity.
  • "1970-03-01" to date is 59.
• Timestamps:
  • LONG and DECIMAL to TIMESTAMP are read as seconds with exact overflow checks: 9223372036854 works and
    9223372036855 is an error.
  • Integer to TIMESTAMP_NTZ is an error.
  • String to timestamp uses spark.sql.session.timeZone.
• Decimal to string strips trailing zeros (-1.00 gives -1). Object or array to string gives the compact JSON
  (Variant.toJson). Doubles print like Java Double.toString, for example 1.0E19. Object or array to any
  other scalar is an error.
• UUID values (from external data) can only be cast to STRING.
• Nested targets:
  • Struct fields are matched by exact name. Missing fields are NULL and extra fields are ignored.
  • Maps take every object field.
  • Arrays cast element by element.
  • variant_get fails on the first bad element. try_variant_get sets only that element to NULL, for example
    Row(1, null).
  • The error's value is the JSON of the failing sub-value ("hello", {}), not the whole input.
• Mix literal and column path arguments, and test v coming from parse_json and from columns.

Differences across Spark versions

• 3.x has no VARIANT.
• 4.0.4, 4.1.3 and 4.2.0 have the same VariantGet, except that 4.2.0 matches CHAR/VARCHAR by type class instead
  of by pattern. The behavior is the same.

Implementation notes

• Spark: VariantGet, VariantGet.cast, VariantPathParser and the builders in
  sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/variantExpressions.scala. Value
  access is in common/variant/src/main/java/org/apache/spark/types/variant/Variant.java (getFieldByKey,
  getElementAtIndex, toJson).
• arrow-rs parquet_variant_compute::variant_get is a different design:
  • Its VariantPath syntax has no $ and treats quoted brackets as fields.
  • Casting follows Arrow CastOptions, not Spark's rules.
  • Its JSON output formats floats differently ({f}, so 1e10 prints as 10000000000).
    Reuse parquet_variant::Variant for decoding and the existing Spark cast kernels in datafusion-spark for
    scalar conversions, but write the path parser, the cast dispatch and toJson to match Spark.
• DataFusion core has no VARIANT support.
• apache/datafusion-comet#5438 tracks native Variant support in Comet.
• Put this in crates/spark/src/function/variant/ with the other VARIANT functions. Because type determines the
  return type, the UDF needs return_field_from_args.
• Add crates/sqllogictest/test_files/variant/variant_get.slt and document both names with #[user_doc].

This issue was generated by an LLM (Claude) from the Apache Spark source code and tests. Examples come from Spark's ExpressionDescription examples, test suites and .sql.out golden files, but please verify them against Spark before relying on them.