Implement Spark-compatible `xpath` (shared XPath 1.0 evaluator for the `xpath_*` family)

Đang mở
#281 0 bình luận 0 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

Chưa có ai nhận issue này.

Đánh giá

Độ khó
5/5
Thời gian dự kiến
Hơn một tuần
Mức phù hợp với người mới
15/100
Loại issue
Tính năng
Độ rõ ràng
Đặc tả rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
rust, xml
Lĩnh vực
databases

Hướng nghiên cứu

Read Spark's UDFXPathUtil.java and xpath.scala alongside the commented query stubs in crates/sqllogictest/test_files/xml/xpath.slt, then look at the empty crates/spark/src/function/xml/mod.rs where the functions must be registered. The first real decision is the XML/XPath dependency (or hand-written evaluator), because the crate has none today, so prototype that against the edge cases listed before writing any function. Done means one shared evaluator backing all nine xpath* functions, xpath.slt re-enabled and passing, and #[user_doc] documentation. Scope spans dependency choice, parser semantics and error messages, so it is not a single-file change.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

enhancement

Part of #271 and apache/datafusion-comet#7300.

What the function does

xpath(xml, path) evaluates an XPath 1.0 expression against an XML string and returns an ARRAY<STRING> with
the DOM node value (Node.getNodeValue()) of every node in the resulting node-set, in document order. Both
arguments are strings. From 4.0 any collation is accepted, including trim collations, and the element type is the
session default string type. path must be foldable. A column path fails analysis with
DATATYPE_MISMATCH.NON_FOLDABLE_INPUT (inputName = `path`). The result is always nullable. Available
since Spark 2.0.0. It was ported from Hive's UDFXPathUtil.

This issue also specifies the shared evaluator behind xpath_boolean, xpath_short, xpath_int,
xpath_long, xpath_float, xpath_double / xpath_number and xpath_string. Those issues cover only their
own result conversion and point back here. All nine functions should share one parser and evaluator.

The node value is the text for text, CDATA and comment nodes and the value for attribute nodes. It is NULL
for element nodes, so a/b returns [null,null,null] while a/b/text() returns the strings.

Examples
SELECT xpath('<a><b>b1</b><b>b2</b><b>b3</b><c>c1</c><c>c2</c></a>','a/b/text()');
-- ["b1","b2","b3"]
SELECT xpath('<a><b>b1</b><b>b2</b><b>b3</b><c>c1</c><c>c2</c></a>','a/b');
-- [null,null,null]
SELECT xpath('<a><b>b1</b><b>b2</b><b>b3</b><c>c1</c><c>c2</c></a>','a/text()');
-- []
SELECT xpath('<a><b>b1</b><b>b2</b><b>b3</b><c>c1</c><c>c2</c></a>','a/*/text()');
-- ["b1","b2","b3","c1","c2"]
SELECT xpath('<a><b class="bb">b1</b><b>b2</b><b>b3</b><c class="bb">c1</c><c>c2</c></a>', 'a/*[@class="bb"]/text()');
-- ["b1","c1"]
SELECT coalesce(xpath('<a></a>', 'a')[0], 'nul');
-- nul

These come from the ExpressionDescription, XPathExpressionSuite and the Hive udf_xpath golden files
that HiveCompatibilitySuite runs.

Shared semantics (all xpath* functions)
  • NULL and empty input. The result is NULL if xml is NULL or '', or if path is NULL or ''
    (XPathEvaluator.evaluate). The check is a plain isEmpty, so a whitespace-only xml is parsed and fails.
  • Lazy path compilation. The path is compiled (and cached) on the first row whose xml is non-empty. An
    invalid path such as '!#$' or 'a/text(' therefore raises nothing when every row is NULL or empty. When it
    does fail, the error is a plain RuntimeException("Invalid XPath '<path>'" + cause message), not a Spark error
    condition.
  • Invalid XML. builder.parse throws a SAXParseException. This is wrapped in
    RuntimeException("Error loading expression '<path>'", e). For '<a>/a>' the root cause message is
    XML document structures must start and end within the same entity. There is no NULL-on-error mode, and
    spark.sql.ansi.enabled has no effect. A result-type conversion failure inside XPathExpression.evaluate
    (an XPathExpressionException) is reported as RuntimeException("Invalid XML document: <msg>\n<xml>").
  • Parser. The parser is the JDK DocumentBuilderFactory with default settings, so it is not
    namespace-aware, does not validate, keeps whitespace-only text nodes, and does not coalesce CDATA. It reads
    from a String reader, so the encoding in an XML declaration is ignored. Only two features are set:
    external-general-entities and external-parameter-entities are both false.
    • The internal DTD subset is processed, so internal entities (<!DOCTYPE a [<!ENTITY e "x">]><a>&e;</a>)
      expand.
    • An external entity reference expands to nothing. The embedFailure test in UDFXPathUtilSuite shows
      xpath_string returning '' for <!ENTITY embed SYSTEM "file">.
    • Loading an external DTD (load-external-dtd) is not disabled. A native implementation should never
      fetch it, and should document that as a known divergence.
  • Namespaces. No NamespaceContext is installed and the DOM is not namespace-aware. A prefixed name test
    such as ns:b cannot be resolved, so the usual workaround is *[local-name()='b']. Spark's tests don't
    cover documents with xmlns, so the behavior of a/b on <a xmlns="urn:x"><b>1</b></a> must be checked
    against Spark before an output is asserted.
  • Engine. This is the JDK's built-in XPath 1.0 (Xalan-derived). Required features include all axes,
    predicates, @attr, //, text(), node(), comment(), unions (|), and the core function library:
    count, sum, string, number, boolean, concat, contains, starts-with, substring*,
    normalize-space, translate, local-name, name, position, last, not, floor, ceiling and
    round. Spark registers no variable or function resolvers, so $var and extension functions fail.
  • Determinism. The functions are deterministic and per-row. Spark marks them null-intolerant, so filters
    get an IsNotNull(xml) (see XPathFunctionsSuite).
Edge cases to test (xpath)
  • NULL xml, NULL path, '' for each, and a non-literal path (analysis error).
  • An empty node-set gives [], not NULL (a/text() above).
  • Element, attribute (a/b/@id), text, comment and CDATA nodes, and mixed content. Element nodes give NULL
    array entries.
  • An expression that doesn't return a node-set, such as count(a/b) or 'x'. By code reading, the JDK can't
    convert the result to NODESET and throws an XPathExpressionException, which Spark reports as
    Invalid XML document: .... Confirm the exact message against Spark.
  • Whitespace-only text nodes between elements: a/node() on <a> <b/> </a> includes them.
  • Unicode text and multi-byte characters in element names.
  • Malformed XML, a DTD with an internal entity, and an external entity (which expands to empty).
Differences across Spark versions
  • 3.4 and 3.5: inputTypes is (StringType, StringType), and the NON_FOLDABLE_INPUT parameter is
    inputName = "path" without backticks. The expression is CodegenFallback with NullIntolerant.
  • 4.0+: the expression is RuntimeReplaceable (an Invoke on an XPathEvaluator). Collated string inputs are
    accepted, and inputName is `path`. Results are unchanged.
  • 4.1 and 4.2: no changes in xpath.scala or UDFXPathUtil.java.
Implementation notes
  • Spark: XPathList in sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/xml/xpath.scala,
    the evaluators in .../expressions/xml/XmlExpressionEvalUtils.scala, and the parser and engine in
    sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/xml/UDFXPathUtil.java.
  • DataFusion core has no XML or XPath functions.
  • crates/spark/src/function/xml/mod.rs exists but registers nothing. The .slt files under
    crates/sqllogictest/test_files/xml/ (xpath.slt, xpath_boolean.slt, xpath_string.slt) only hold
    commented-out queries ported from Sail, and should be enabled.
  • The crate has no XML dependency today, so one must be added. Options:
    • sxd-document + sxd-xpath: the only pure-Rust XPath 1.0 engine. The last release was 2019 (0.4.x).
      Check its DTD and internal-entity support, its number-to-string formatting, its NaN handling, and its
      error behavior against Xalan.
    • roxmltree or quick-xml + a hand-written XPath 1.0 evaluator. roxmltree expands internal entities and
      does not resolve external ones. Neither crate ships an XPath engine.
    • xee-xpath targets XPath 3.1. Its typing, comparisons, sum() of non-numeric strings and number
      formatting differ from 1.0, so it would need a 1.0 compatibility layer.
    • libxml (libxml2 bindings) has complete XPath 1.0 but adds a C dependency.
      Whichever is chosen, the XPath 1.0 behaviors to verify are: string-to-number (no exponent, no leading +,
      surrounding whitespace allowed, anything else NaN); number-to-string (3, not 3.0; NaN, Infinity);
      node-set comparisons with existential semantics; and the string value of an element (concatenated
      descendant text).
  • Compile the path once per batch, since it is a literal, and return an error for a non-scalar path.
  • Tests: add crates/sqllogictest/test_files/xml/xpath.slt (replace the commented stub) and document the
    function with #[user_doc].

This issue was generated by an LLM (Claude) from the Apache Spark source code and tests. Examples come from Spark's ExpressionDescription examples, test suites and .sql.out golden files, but please verify them against Spark before relying on them.

Ngôn ngữ chính
Rust
Star
7
Fork
15
Merge trung bình
16 giờ 43 phút
Pull request đã merge (30 ngày)
26

Chuẩn bị môi trường

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của apache/datafusion-spark

Tất cả issue của apache/datafusion-spark

Issue tương tự

Thêm issue về Rust

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.