Implement Spark-compatible `regexp_extract_all`

Đang mở
  • Có gì không ổn?
#257 1 bình luận 0 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

Chưa có ai nhận issue này.

Đánh giá

Độ khó
5/5
Thời gian dự kiến
Hơn một tuần
Mức phù hợp với người mới
22/100
Loại issue
Tính năng
Độ rõ ràng
Đặc tả rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
rust, spark, sql
Lĩnh vực
backend, databases

Hướng nghiên cứu

Start by reading RegExpExtractAll and RegExpExtractBase.checkGroupIndex in Spark's regexpExpressions.scala, then look at an existing string function under crates/spark/src/function/string/ to copy the registration, #[user_doc] and error-reporting patterns. This depends on the shared Java-regex compatibility helper proposed for rlike, so check its status before writing matching logic; the empty-match iteration rule and group index errors are the parts that must be reproduced exactly. Done means a new function plus crates/sqllogictest/test_files/string/regexp_extract_all.slt covering the NULL, no-match, out-of-range idx and optional-group cases above.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

enhancement

Part of #174 and apache/datafusion-comet#7300.

What the function does

regexp_extract_all(str, regexp[, idx]) returns an array with group idx of every match of the Java regular expression regexp in str. idx defaults to 1; 0 means the whole match. Spark loops Matcher.find() and, for each match, checks 0 <= idx <= groupCount and appends group(idx), or '' when the group did not participate. Available since 3.1.0.

str and regexp are implicitly cast to STRING and idx to INT. idx may be a column. Result type is ARRAY<STRING> (element type follows str's collation in 4.0+, containsNull = true but elements are never NULL). NULL if any argument is NULL. No match returns an empty array.

Examples
SELECT regexp_extract_all('100-200, 300-400', '(\\d+)-(\\d+)', 1);
-- ["100","300"]
SELECT regexp_extract_all('1a 2b 14m', '(\\d+)([a-z]+)', 0);
-- ["1a","2b","14m"]
SELECT regexp_extract_all('1a 2b 14m', '(\\d+)([a-z]+)', 2);
-- ["a","b","m"]
SELECT regexp_extract_all('a 2b 14m', '(\\d+)?([a-z]+)', 1);
-- ["","2","14"]
SELECT regexp_extract_all('1a 2b 14m', '(\\d+)?', 1);
-- ["1","","","2","","","14","",""]
SELECT regexp_extract_all('1a 2b 14m', '\\d+');
-- error INVALID_PARAMETER_VALUE.REGEX_GROUP_INDEX (groupCount 0, groupIndex 1)
SELECT regexp_extract_all('abc', col0, 1) FROM VALUES('], [') AS t(col0);
-- error INVALID_PARAMETER_VALUE.PATTERN
Edge cases to test
  • NULL str, NULL regexp, NULL idx (literal and column): NULL.
  • No match: []. regexp_extract_all('100-200,300-400,500-600', '([a-z])', 1) is [] (RegexpExpressionsSuite).
  • Group index validation only happens inside the match loop, so an out-of-range idx raises no error when there is no match: regexp_extract_all('xyz', '\\d+', 1) returns [].
  • Out-of-range or negative idx with a match: INVALID_PARAMETER_VALUE.REGEX_GROUP_INDEX (SQLSTATE 22023), message The value of parameter(s) `idx` in `regexp_extract_all` is invalid: Expects group index between 0 and 2, but got 3.
  • Optional group that did not participate yields '', not NULL (see the (\\d+)?([a-z]+) example).
  • Empty pattern: matches at every position, so regexp_extract_all('abc', '', 0) returns four empty strings.
  • Empty matches after a non-empty match: the (\\d+)? example above has 9 elements because Java reports the empty match at the end of each \d+ match. The regex crate does not report an empty match that starts where the previous match ended (its testdata/empty.toml gives 3 matches for b| on abc, Java finds 4), so a direct captures_iter port returns fewer elements. Iteration must follow Java's rule.
  • Invalid pattern: INVALID_PARAMETER_VALUE.PATTERN with value the quoted pattern, also for per-row patterns ('] [', '], [').
  • Non-literal pattern column with different patterns per row (Spark caches the last regex).
  • spark.sql.parser.escapedStringLiterals (default false) changes how the SQL literal is unescaped; the kernel sees the result. In .slt, DataFusion keeps backslashes literal ('(\d+)').
  • Java vs Rust pattern divergences to decline or test: lookaround / backreferences / atomic / possessive (Java only; e.g. (?=[^ ]*[0-9])[^ ]+); ASCII \d \w \s in Java vs Unicode in Rust (regexp_extract_all('1\u0663', '\\d', 0) (U+0663 ARABIC-INDIC DIGIT THREE) is ["1"] in Spark); . excludes \r, \u0085, \u2028, \u2029 in Java; $ matches before a final line terminator in Java; (?i) ASCII-only in Java; inline flag letters U / u / d mean different things or are unknown; named groups: Java (?<n>...) only, Rust also (?P<n>...); both number named groups in order, so groupCount matches for accepted patterns.
  • Unicode / supplementary characters in str (Java iterates UTF-16 but never splits a surrogate pair for .).
  • UTF8_LCASE collation (4.0+) compiles with CASE_INSENSITIVE | UNICODE_CASE.
Differences across Spark versions
  • 3.4: group index errors were _LEGACY_ERROR_TEMP_2006 (The specified group index cannot be less than zero.) and _LEGACY_ERROR_TEMP_2007 (Regex group count is <groupCount>, but the specified group index is <groupIndex>.), both SparkIllegalArgumentException. 3.5+ uses INVALID_PARAMETER_VALUE.REGEX_GROUP_INDEX for both cases.
  • 4.0: collation-aware inputs; array element type follows str.
Implementation notes
  • Spark: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/regexpExpressions.scala, RegExpExtractAll (base RegExpExtractBase, RegExpExtractBase.checkGroupIndex).
  • DataFusion core has no equivalent. regexp_match returns the capture groups of the first match only, uses Rust syntax, and returns NULL on no match.
  • Use the shared Java-regex compatibility helper proposed for rlike: accept only a verified subset, return an "unsupported pattern" error otherwise (at planning time for literal patterns), and implement Java's empty-match iteration rule. regexp_count and regexp_substr are rewritten by Spark into this function and regexp_extract, so the same core can back them.
  • Belongs in crates/spark/src/function/string/. Add crates/sqllogictest/test_files/string/regexp_extract_all.slt and document with #[user_doc].

This issue was generated by an LLM (Claude) from the Apache Spark source code and tests. Examples come from Spark's ExpressionDescription examples, test suites and .sql.out golden files, but please verify them against Spark before relying on them.

Ngôn ngữ chính
Rust
Star
7
Fork
15
Merge trung bình
16 giờ 43 phút
Pull request đã merge (30 ngày)
26

Chuẩn bị môi trường

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của apache/datafusion-spark

Tất cả issue của apache/datafusion-spark

Issue tương tự

Thêm issue về Rust

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.