Implement Spark-compatible `schema_of_xml`

Đang mở
#292 0 bình luận 0 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

Chưa có ai nhận issue này.

Đánh giá

Độ khó
4/5
Thời gian dự kiến
3-5 ngày
Mức phù hợp với người mới
42/100
Loại issue
Tính năng
Độ rõ ràng
Đặc tả rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
rust, sql, xml
Lĩnh vực
backend

Hướng nghiên cứu

Start by reading the existing from_xml implementation under crates/spark/src/function/xml/ and its tests, then compare against Spark's SchemaOfXml and XmlInferSchema referenced in the issue. Add the new function in the same XML module, reusing the tokenizer and XML options handling, plus a Spark-style DataType.sql printer, and exercise it through the new crates/sqllogictest/test_files/xml/schema_of_xml.slt. Done means the listed examples and edge cases (foldability, malformed modes, case sensitivity, value inference) produce Spark-matching output and the docs are generated via #[user_doc].

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

enhancement

Part of #271 and apache/datafusion-comet#7300.

What the function does

schema_of_xml(xml[, options]) infers the Spark type of one XML record and returns it as a DDL-formatted
STRING (DataType.sql), for example STRUCT<a: BIGINT, b: STRING>. It was added in Spark 4.0.0.

  • Arguments. xml must be foldable. A column fails with DATATYPE_MISMATCH.NON_FOLDABLE_INPUT, and
    schema_of_xml(null) fails with DATATYPE_MISMATCH.UNEXPECTED_NULL. options is a map(...) of string
    literals using the XmlOptions keys.
  • Result. The result is non-nullable and in practice a constant, typically passed as the schema of
    from_xml.
  • Root element. Its name is ignored. Its attributes and children become the fields.

Inference is XmlInferSchema.infer(xml) followed by canonicalizeType, with XmlOptions in the UTC time
zone and case sensitivity from spark.sql.caseSensitive. A StructType result whose fields are all removed
by canonicalization becomes STRUCT<>.

Examples
SELECT schema_of_xml('<p><a>1</a></p>');
-- STRUCT<a: BIGINT>
SELECT schema_of_xml('<p><a attr="2">1</a><a>3</a></p>', map('excludeAttribute', 'true'));
-- STRUCT<a: ARRAY<BIGINT>>
SELECT schema_of_xml('<p><a>1</a><b>"2"</b></p>');
-- STRUCT<a: BIGINT, b: STRING>
SELECT schema_of_xml('<record><time>14:30:45</time></record>', map('rowTag', 'record'));
-- STRUCT<time: TIMESTAMP>
SELECT schema_of_xml(regexp_replace('<ROW><item_id>1</item_id><item_price>0.1</item_price></ROW>', 'item_', ''));
-- STRUCT<id: BIGINT, price: DOUBLE>
SELECT schema_of_xml('<ROW><id></id></ROW>');
-- STRUCT<id: STRING>
SELECT schema_of_xml('<ROW><a>1<ROW>', map('mode', 'PERMISSIVE'));
-- STRUCT<_corrupt_record: STRING>
SELECT schema_of_xml(42);
-- error: DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE (4.1+)

These come from the ExpressionDescription, xml-functions.sql.out, XmlFunctionsSuite and XmlSuite. The
regexp_replace case shows that any foldable expression is accepted, not only a literal.

Edge cases to test
  • Value inference (inferFrom, after trimming when ignoreSurroundingSpaces=true, the default). The rules
    are tried in this order:

    1. empty: NullType, which becomes STRING;
    2. Long-parseable (optional sign, first character a digit): BIGINT;
    3. with prefersDecimal=true, decimal-parseable: DECIMAL(max(p, s), s);
    4. double-parseable, but not ending in d, D, f or F: DOUBLE;
    5. true or false, case-insensitive: BOOLEAN;
    6. a date per dateFormat, only when preferDate=true (the default, and forced off under the legacy time
      parser policy): DATE;
    7. TIMESTAMP_NTZ, only when spark.sql.timestampType=TIMESTAMP_NTZ or the legacy policy is on;
    8. a timestamp per timestampFormat, or the default parser: TIMESTAMP (14:30:45 above);
    9. anything else: STRING.

    With inferSchema=false, every value is STRING.

  • Structure.

    • Elements become struct fields sorted by name.
    • A repeated element becomes ARRAY<...>, with the element types merged by compatibleType. That function
      uses findTightestCommonType and has these rules:
      • DOUBLE with DECIMAL gives DOUBLE;
      • a decimal that would need precision above 38 becomes DOUBLE;
      • TIMESTAMP_NTZ with TIMESTAMP gives TIMESTAMP;
      • structs merge their fields;
      • an array with a scalar gives an array;
      • a struct with a valueTag field merged with a scalar widens the value field;
      • anything else gives STRING.
    • Attributes become _-prefixed fields (attributePrefix). An element with attributes and text gets a
      _VALUE field (valueTag).
  • Case sensitivity. With spark.sql.caseSensitive=false, <foo>, <Foo> and <FOO> merge into one array
    field named after the first name seen. With true, they are three fields.

  • Other options. ignoreNamespace, excludeAttribute, nullValue, dateFormat, timestampFormat,
    timestampNTZFormat and locale.

  • Malformed input.

    • PERMISSIVE returns STRUCT<_corrupt_record: STRING>, using columnNameOfCorruptRecord or the
      spark.sql.columnNameOfCorruptRecord config.
    • FAILFAST raises MALFORMED_RECORD_IN_PARSING.WITHOUT_SUGGESTION, and badRecord is the corrupt column
      name (_corrupt_record), not the input.
    • DROPMALFORMED raises PARSE_MODE_UNSUPPORTED with funcName `schema_of_xml`.
  • Other inputs. Also test an empty element (STRING), deeply nested structs, mixed content, and Unicode
    element names.

Differences across Spark versions
  • Not available in 3.4 or 3.5.
  • 4.1 adds the UNEXPECTED_INPUT_TYPE check for a non-STRING foldable argument (schema_of_xml(42)). In 4.0
    that query has no golden result.
  • 4.2: TIME values are still inferred as TIMESTAMP (golden above). There is no TIME inference.
Implementation notes
  • Spark: SchemaOfXml in sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/xmlExpressions.scala
    (RuntimeReplaceable, a StaticInvoke of XmlExpressionEvalUtils.schemaOfXml), with inference in
    sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/xml/XmlInferSchema.scala.
  • DataFusion core has no equivalent.
  • The argument is a literal, so the kernel can run once (at planning time or on the first batch) and return a
    scalar. It needs a Spark-style DataType.sql printer (STRUCT<a: BIGINT>, upper-case, : separators).
    Share it with schema_of_json and schema_of_csv if they exist by then.
  • Use the same XML tokenizer as from_xml (quick-xml is a natural fit). The crate has no XML dependency
    today. Put it in crates/spark/src/function/xml/.
  • Tests: add crates/sqllogictest/test_files/xml/schema_of_xml.slt and document the function with #[user_doc].

This issue was generated by an LLM (Claude) from the Apache Spark source code and tests. Examples come from Spark's ExpressionDescription examples, test suites and .sql.out golden files, but please verify them against Spark before relying on them.

Ngôn ngữ chính
Rust
Star
7
Fork
15
Merge trung bình
16 giờ 43 phút
Pull request đã merge (30 ngày)
26

Chuẩn bị môi trường

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của apache/datafusion-spark

Tất cả issue của apache/datafusion-spark

Issue tương tự

Thêm issue về Rust

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.