Implement Spark-compatible `parse_json` / `try_parse_json` (and a Spark VARIANT representation)

Start by reading Spark's VariantBuilder.java and VariantUtil.java alongside common/variant/README.md to pin down the byte layout, then check arrow-rs parquet-variant and parquet-variant-compute at the arrow 60 versions for what can be reused for decoding. The new encoder belongs in crates/spark/src/function/variant/, with parse_json/try_parse_json registered as UDFs and the extension set in return_field_from_args. Done means crates/sqllogictest/test_files/variant/parse_json.slt passes on the examples and edge cases listed here, with to_json available to display values.

Part of #273 and apache/datafusion-comet#7300.

What the function does

parse_json(jsonStr) parses a JSON string into a VARIANT value. If the input is not valid JSON, it raises an
error. try_parse_json(jsonStr) does the same but returns NULL for malformed input, duplicate keys, and the size
limit. Both take exactly one STRING argument (any collation, including trim collations). SQL NULL input
returns SQL NULL. parse_json is non-nullable for non-null input and try_parse_json is nullable. Both were
added in Spark 4.0.0 (variant_funcs). Spark 3.x has no VARIANT type.

This is the first VARIANT function, so this issue also covers how datafusion-spark represents the type. Every
other variant function (variant_get, is_variant_null, schema_of_variant, schema_of_variant_agg,
to_variant_object) needs that representation.

A Spark VARIANT is two binaries. Spark implements the Parquet VariantEncoding.md spec as of parquet-format
commit 37b6e8b8 (see common/variant/README.md):

• metadata: a header byte (version 1 in the low 4 bits, bit 4 = sorted_strings, bits 6-7 =
  offset_size - 1), the dictionary size, size + 1 offsets, then the UTF-8 key bytes.
• value: a header byte whose low 2 bits are the basic type (primitive, short string, object, array) and whose
  upper 6 bits are type info, followed by the content.

Examples

SELECT parse_json('{"a":1,"b":0.8}');
-- {"a":1,"b":0.8}
SELECT try_parse_json('{"a":1,');
-- NULL
SELECT to_json(parse_json('[0.0, 1.00, 1.10, 1.23]'));
-- [0,1,1.1,1.23]
SELECT to_json(parse_json('1.0E10'));
-- 1.0E10
SELECT parse_json('{"a": 1, "b": 2, "a": "3"}');
-- error MALFORMED_RECORD_IN_PARSING.WITHOUT_SUGGESTION (cause: VARIANT_DUPLICATE_KEY)
SELECT try_parse_json('{"a": 1, "b": 2, "a": "3"}');
-- NULL
SET spark.sql.variant.allowDuplicateKeys=true;
SELECT to_json(parse_json('{"c": [], "b": 0, "a": null, "a": {"x": 0, "x": 1}, "b": 1, "b": 2, "c": [3]}'));
-- {"a":{"x":1},"b":2,"c":[3]}
SELECT try_parse_json('{1:2}'), try_parse_json('{"a":[a,b,c]}');
-- NULL  NULL

These come from the ExpressionDescription, VariantEndToEndSuite and VariantExpressionEvalUtilsSuite.

Edge cases to test

Byte-level expectations are in VariantExpressionEvalUtilsSuite."parseJson positive". Port them as unit tests.

• Integers use the smallest of INT1, INT2, INT4 and INT8: 127 is INT1, 128 is INT2, -32769 is INT4 and
  2147483648 is INT8. An integer outside i64 (for example 9223372036854775808) becomes DECIMAL16 with scale
  0 if it has at most 38 digits. Otherwise it becomes DOUBLE (1 followed by 38 zeros is DOUBLE 1E38).
• Non-integers become a decimal only if the text contains just -, . and digits and fits precision/scale 38.
  The original scale is kept: 1.0 is stored as DECIMAL4 with scale 1 and unscaled 10. DECIMAL4 is used when
  precision and scale are both <= 9, DECIMAL8 when they are <= 18, and DECIMAL16 otherwise (99999.9999 is
  DECIMAL4, 99999.99999 is DECIMAL8). Anything with an exponent (1e10, 1E0) is DOUBLE, and so are values
  over 38 digits.
• Strings of up to 63 UTF-8 bytes use the short-string header. Longer strings use LONG_STR with a 4-byte length.
• Objects:
  • Field ids and offsets are sorted by key, but the field data stays in document order. For
    {"z": 1, "y": 2, "x": "3"} the ids are 2,1,0 and the offsets are 4,2,0,6.
  • Keys are compared with Java String.compareTo, which is UTF-16 code-unit order, not UTF-8 byte order. Test
    keys that mix supplementary characters (U+10000 and above) with U+E000-U+FFFF.
  • Id and offset widths are the minimum that fits, and the size field becomes 4 bytes above 255 fields.
• The metadata dictionary is in first-seen document order, without duplicates. sorted_strings is never set,
  and the offset width is the minimum that fits. Spark writes {} and scalars with metadata [0x01, 0, 0].
• Duplicate keys:
  • spark.sql.variant.allowDuplicateKeys is internal and defaults to false. The VARIANT_DUPLICATE_KEY error
    is wrapped as MALFORMED_RECORD_IN_PARSING.WITHOUT_SUGGESTION with badRecord set to the input and
    failFastMode = FAILFAST.
  • With true, the last occurrence wins and the dead bytes are compacted.
• Malformed input such as "", [, +1, 1a, single quotes, NaN and leading zeros gives the same
  MALFORMED_RECORD_IN_PARSING.WITHOUT_SUGGESTION error. Spark uses a default Jackson JsonFactory, so all the
  lenient features are off.
• Trailing content: Spark calls nextToken() once and never checks for end of input. By code reading,
  '{"a":1} x' may parse successfully. Confirm this against Spark before asserting it.
• Size limit: value and metadata are each limited to 128 MiB (16 MiB under SPARK_TESTING). Going over raises
  VARIANT_SIZE_LIMIT ("Cannot build variant bigger than 128.0 MiB in parse_json."), and try_parse_json
  returns NULL. Jackson's StreamReadConstraints defaults (string length, number length, nesting depth) can fail
  first, with the malformed-record error instead.
• Lone UTF-16 surrogates such as "\uD835" and {"\uDC00":1} are rejected with the malformed-record error when
  spark.sql.variant.validateUnicodeInJsonParsing is true (internal, default true). With false they are
  silently replaced by U+FFFD. Valid surrogate pairs are accepted.
• Test scalar and column inputs, NULL, the empty string, whitespace-only input, and deep nesting.

Differences across Spark versions

• 3.4 and 3.5 have no VARIANT.
• 4.0.4, 4.1.3 and 4.2.0 share the same ParseJson and builder logic.
• Only the 4.2.0 tree checked here has the lone-surrogate validation and its config (tagged .version("4.3.0")).
  4.0.4 and 4.1.3 accept lone surrogates and replace them with U+FFFD.
• Jackson is 2.18.6 in 4.0 and 2.21.2 in 4.1 and 4.2. That can change the parser constraint limits.

Implementation notes

• Spark: ParseJson and ParseJsonExpressionBuilder / TryParseJsonExpressionBuilder in
  sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/variantExpressions.scala.
  VariantExpressionEvalUtils.parseJson is in the same package. The encoder is
  common/variant/src/main/java/org/apache/spark/types/variant/VariantBuilder.java, and the constants and
  SIZE_LIMIT are in VariantUtil.java.
• Arrow representation:
  • Use the canonical Arrow extension type arrow.parquet.variant, which arrow-rs implements as
    parquet_variant_compute::VariantType. It is a Struct with metadata and value binary children (Binary,
    LargeBinary or BinaryView), plus an optional typed_value child when shredded.
  • Spark's own ArrowUtils maps VariantType to Struct<value: Binary not null, metadata: Binary not null>.
    The metadata child carries field metadata variant=true and there is no extension name. Engines that
    exchange Arrow with Spark have to translate between the two.
  • The UDF sets the extension in return_field_from_args.
• arrow-rs has parquet-variant, parquet-variant-compute and parquet-variant-json, but its encoding is not
  byte-identical to Spark's. The local arrow-rs is 58.2. datafusion-spark is on arrow 60, so use the matching
  crate versions.
  • parquet-variant-json::append_json uses serde_json. Non-integers and integers outside i64 become DOUBLE,
    never decimal.
  • Duplicate keys silently keep the last value.
  • Keys are visited in serde_json::Map order, which is sorted unless the preserve_order feature is unified in.
    That changes the dictionary order.
  • serde_json stops at nesting depth 128 and rejects trailing content.
  • WritableMetadataBuilder sets sorted_strings when keys happen to arrive in sorted order, including any
    single-key dictionary. Spark never sets it.
  • ObjectBuilder::finish sorts by Rust str byte order, not UTF-16 order.
  • When duplicates are allowed, the old bytes are left in place, so sizes differ.
  • I found no size limit in the builder.
• Spark reads any valid encoding (VariantExpressionSuite."to_json valid input"), so readers only need logical
  equivalence. Field order is the exception. Spark's schema_of_variant validates sorting with UTF-16 compare
  and throws MALFORMED_VARIANT, and getFieldByKey binary-searches objects with 32 or more fields. A writer
  that sorts by UTF-8 bytes can therefore produce values that Spark rejects. The size limit also depends on
  Spark's exact byte layout.
• Recommendation:
  • Reuse parquet_variant::Variant for decoding.
  • Write a Spark-specific JSON-to-variant encoder in a new crates/spark/src/function/variant/ module, or a
    configured VariantBuilder with Spark's number classification, document-order dictionary, UTF-16 key
    ordering, duplicate handling, cleared sorted_strings and the 128 MiB limit.
  • Use a JSON tokenizer that exposes the raw number text.
• DataFusion core has no VARIANT support or parse_json.
• apache/datafusion-comet#5438 tracks native Variant support in Comet.
• Add crates/sqllogictest/test_files/variant/parse_json.slt. It will need to_json or a cast to string to
  display values. Document both names with #[user_doc].

This issue was generated by an LLM (Claude) from the Apache Spark source code and tests. Examples come from Spark's ExpressionDescription examples, test suites and .sql.out golden files, but please verify them against Spark before relying on them.