Implement Spark-compatible `xpath` (shared XPath 1.0 evaluator for the `xpath_*` family)
维护者通常 1 天内回复
还没有人认领这个 Issue。
评估
调研方向
Read Spark's UDFXPathUtil.java and xpath.scala alongside the commented query stubs in crates/sqllogictest/test_files/xml/xpath.slt, then look at the empty crates/spark/src/function/xml/mod.rs where the functions must be registered. The first real decision is the XML/XPath dependency (or hand-written evaluator), because the crate has none today, so prototype that against the edge cases listed before writing any function. Done means one shared evaluator backing all nine xpath* functions, xpath.slt re-enabled and passing, and #[user_doc] documentation. Scope spans dependency choice, parser semantics and error messages, so it is not a single-file change.
由索引模型根据 Issue 内容生成。
描述
Part of #271 and apache/datafusion-comet#7300.
What the function does
xpath(xml, path) evaluates an XPath 1.0 expression against an XML string and returns an ARRAY<STRING> with
the DOM node value (Node.getNodeValue()) of every node in the resulting node-set, in document order. Both
arguments are strings. From 4.0 any collation is accepted, including trim collations, and the element type is the
session default string type. path must be foldable. A column path fails analysis with
DATATYPE_MISMATCH.NON_FOLDABLE_INPUT (inputName = `path`). The result is always nullable. Available
since Spark 2.0.0. It was ported from Hive's UDFXPathUtil.
This issue also specifies the shared evaluator behind xpath_boolean, xpath_short, xpath_int,
xpath_long, xpath_float, xpath_double / xpath_number and xpath_string. Those issues cover only their
own result conversion and point back here. All nine functions should share one parser and evaluator.
The node value is the text for text, CDATA and comment nodes and the value for attribute nodes. It is NULL
for element nodes, so a/b returns [null,null,null] while a/b/text() returns the strings.
Examples
SELECT xpath('<a><b>b1</b><b>b2</b><b>b3</b><c>c1</c><c>c2</c></a>','a/b/text()');
-- ["b1","b2","b3"]
SELECT xpath('<a><b>b1</b><b>b2</b><b>b3</b><c>c1</c><c>c2</c></a>','a/b');
-- [null,null,null]
SELECT xpath('<a><b>b1</b><b>b2</b><b>b3</b><c>c1</c><c>c2</c></a>','a/text()');
-- []
SELECT xpath('<a><b>b1</b><b>b2</b><b>b3</b><c>c1</c><c>c2</c></a>','a/*/text()');
-- ["b1","b2","b3","c1","c2"]
SELECT xpath('<a><b class="bb">b1</b><b>b2</b><b>b3</b><c class="bb">c1</c><c>c2</c></a>', 'a/*[@class="bb"]/text()');
-- ["b1","c1"]
SELECT coalesce(xpath('<a></a>', 'a')[0], 'nul');
-- nul
These come from the ExpressionDescription, XPathExpressionSuite and the Hive udf_xpath golden files
that HiveCompatibilitySuite runs.
Shared semantics (all xpath* functions)
- NULL and empty input. The result is NULL if
xmlis NULL or'', or ifpathis NULL or''
(XPathEvaluator.evaluate). The check is a plainisEmpty, so a whitespace-onlyxmlis parsed and fails. - Lazy path compilation. The path is compiled (and cached) on the first row whose
xmlis non-empty. An
invalid path such as'!#$'or'a/text('therefore raises nothing when every row is NULL or empty. When it
does fail, the error is a plainRuntimeException("Invalid XPath '<path>'" + cause message), not a Spark error
condition. - Invalid XML.
builder.parsethrows aSAXParseException. This is wrapped in
RuntimeException("Error loading expression '<path>'", e). For'<a>/a>'the root cause message is
XML document structures must start and end within the same entity.There is no NULL-on-error mode, and
spark.sql.ansi.enabledhas no effect. A result-type conversion failure insideXPathExpression.evaluate
(anXPathExpressionException) is reported asRuntimeException("Invalid XML document: <msg>\n<xml>"). - Parser. The parser is the JDK
DocumentBuilderFactorywith default settings, so it is not
namespace-aware, does not validate, keeps whitespace-only text nodes, and does not coalesce CDATA. It reads
from aStringreader, so theencodingin an XML declaration is ignored. Only two features are set:
external-general-entitiesandexternal-parameter-entitiesare both false.- The internal DTD subset is processed, so internal entities (
<!DOCTYPE a [<!ENTITY e "x">]><a>&e;</a>)
expand. - An external entity reference expands to nothing. The
embedFailuretest inUDFXPathUtilSuiteshows
xpath_stringreturning''for<!ENTITY embed SYSTEM "file">. - Loading an external DTD (
load-external-dtd) is not disabled. A native implementation should never
fetch it, and should document that as a known divergence.
- The internal DTD subset is processed, so internal entities (
- Namespaces. No
NamespaceContextis installed and the DOM is not namespace-aware. A prefixed name test
such asns:bcannot be resolved, so the usual workaround is*[local-name()='b']. Spark's tests don't
cover documents withxmlns, so the behavior ofa/bon<a xmlns="urn:x"><b>1</b></a>must be checked
against Spark before an output is asserted. - Engine. This is the JDK's built-in XPath 1.0 (Xalan-derived). Required features include all axes,
predicates,@attr,//,text(),node(),comment(), unions (|), and the core function library:
count,sum,string,number,boolean,concat,contains,starts-with,substring*,
normalize-space,translate,local-name,name,position,last,not,floor,ceilingand
round. Spark registers no variable or function resolvers, so$varand extension functions fail. - Determinism. The functions are deterministic and per-row. Spark marks them null-intolerant, so filters
get anIsNotNull(xml)(seeXPathFunctionsSuite).
Edge cases to test (xpath)
- NULL
xml, NULLpath,''for each, and a non-literal path (analysis error). - An empty node-set gives
[], not NULL (a/text()above). - Element, attribute (
a/b/@id), text, comment and CDATA nodes, and mixed content. Element nodes give NULL
array entries. - An expression that doesn't return a node-set, such as
count(a/b)or'x'. By code reading, the JDK can't
convert the result toNODESETand throws anXPathExpressionException, which Spark reports as
Invalid XML document: .... Confirm the exact message against Spark. - Whitespace-only text nodes between elements:
a/node()on<a> <b/> </a>includes them. - Unicode text and multi-byte characters in element names.
- Malformed XML, a DTD with an internal entity, and an external entity (which expands to empty).
Differences across Spark versions
- 3.4 and 3.5:
inputTypesis(StringType, StringType), and theNON_FOLDABLE_INPUTparameter is
inputName = "path"without backticks. The expression isCodegenFallbackwithNullIntolerant. - 4.0+: the expression is
RuntimeReplaceable(anInvokeon anXPathEvaluator). Collated string inputs are
accepted, andinputNameis`path`. Results are unchanged. - 4.1 and 4.2: no changes in
xpath.scalaorUDFXPathUtil.java.
Implementation notes
- Spark:
XPathListinsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/xml/xpath.scala,
the evaluators in.../expressions/xml/XmlExpressionEvalUtils.scala, and the parser and engine in
sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/xml/UDFXPathUtil.java. - DataFusion core has no XML or XPath functions.
crates/spark/src/function/xml/mod.rsexists but registers nothing. The.sltfiles under
crates/sqllogictest/test_files/xml/(xpath.slt,xpath_boolean.slt,xpath_string.slt) only hold
commented-out queries ported from Sail, and should be enabled.- The crate has no XML dependency today, so one must be added. Options:
sxd-document+sxd-xpath: the only pure-Rust XPath 1.0 engine. The last release was 2019 (0.4.x).
Check its DTD and internal-entity support, its number-to-string formatting, itsNaNhandling, and its
error behavior against Xalan.roxmltreeorquick-xml+ a hand-written XPath 1.0 evaluator.roxmltreeexpands internal entities and
does not resolve external ones. Neither crate ships an XPath engine.xee-xpathtargets XPath 3.1. Its typing, comparisons,sum()of non-numeric strings and number
formatting differ from 1.0, so it would need a 1.0 compatibility layer.libxml(libxml2 bindings) has complete XPath 1.0 but adds a C dependency.
Whichever is chosen, the XPath 1.0 behaviors to verify are: string-to-number (no exponent, no leading+,
surrounding whitespace allowed, anything elseNaN); number-to-string (3, not3.0;NaN,Infinity);
node-set comparisons with existential semantics; and the string value of an element (concatenated
descendant text).
- Compile the path once per batch, since it is a literal, and return an error for a non-scalar path.
- Tests: add
crates/sqllogictest/test_files/xml/xpath.slt(replace the commented stub) and document the
function with#[user_doc].
This issue was generated by an LLM (Claude) from the Apache Spark source code and tests. Examples come from Spark's ExpressionDescription examples, test suites and .sql.out golden files, but please verify them against Spark before relying on them.
- 主要语言
- Rust
- 星标
- 7
- 派生
- 15
- 平均合并
- 16 小时 43 分钟
- 30 天内合并 PR
- 26
环境准备
- 没有 Dockerfile 或 Docker Compose 文件
- 没有 Pull Request 模板
- 阅读贡献指南
从这里开始
- 先读完整个 Issue,再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库,在一个分支上完成修改。
- 提交 Pull Request,并在描述里引用这个 Issue 编号。
apache/datafusion-spark 的其他 Issue
-
enhancement
难度 2/5 1-3 小时 新手友好度 70/100
apache/datafusion-spark#287 ·
维护者通常 1 天内回复
-
enhancement good first issue
难度 2/5 1-3 小时 新手友好度 80/100
apache/datafusion-spark#266 ·
维护者通常 1 天内回复
-
[Bug] make_dt_interval returns NULL for more than about 1.49 million days, Spark returns the interval可能已有人在做 @Subodh-17 今天认领。 未关闭
难度 2/5 1-3 小时 新手友好度 78/100
apache/datafusion-spark#109 ·
维护者通常 1 天内回复
-
enhancement
难度 5/5 一周以上 新手友好度 18/100
apache/datafusion-spark#299 ·
维护者通常 1 天内回复
-
enhancement
难度 4/5 3-5 天 新手友好度 35/100
apache/datafusion-spark#298 ·
维护者通常 1 天内回复
查看 apache/datafusion-spark 的全部 Issue
相似的 Issue
-
comp-rbac
难度 2/5 1-3 小时 新手友好度 68/100
ClickHouse/ClickHouse#131875 ·
维护者通常 1 天内回复
-
bug(python): read_transaction raises TypeError on a Rewrite with rewritten indices可能已有人在做 @Borg93 今天认领。 未关闭bug
难度 2/5 1-3 小时 新手友好度 75/100
lance-format/lance#11117 ·
维护者通常 2 天内回复
-
难度 2/5 1-3 小时 新手友好度 70/100
storytold/photocraft#15232 · 1 条评论 ·
维护者通常 1 天内回复
-
forge: fork preflight reads still send `requireCanonical: false`, which Rootstock rejects (-32603)可能已有人在做 @Ynyesto 今天认领。 未关闭
难度 2/5 1-3 小时 新手友好度 72/100
foundry-rs/foundry#18485 ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 70/100
bondybondbond/rushcut#238 · 1 条评论 ·