Implement Spark-compatible `split`

Đang mở
  • Có gì không ổn?
#261 1 bình luận 0 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

Chưa có ai nhận issue này.

Đánh giá

Độ khó
4/5
Thời gian dự kiến
3-5 ngày
Mức phù hợp với người mới
20/100
Loại issue
Tính năng
Độ rõ ràng
Đặc tả rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
rust, spark
Lĩnh vực
backend

Hướng nghiên cứu

Start by reading the existing string functions under crates/spark/src/function/string/ to see the registration and #[user_doc] pattern, then check whether the Java-regex compatibility helper from the rlike work exists yet — this issue depends on it. Read Spark's StringSplit and UTF8String.split for the leading-empty and limit rules, since Regex::split cannot be used directly. Done means split.slt under crates/sqllogictest/test_files/string/ passes the NULL, limit, empty-regex and column-argument cases listed here.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

enhancement

Part of #174 and apache/datafusion-comet#7300.

What the function does

split(str, regex[, limit]) splits str around matches of the Java regular expression regex and returns an array. limit (default -1) controls how many times the regex is applied: if limit > 0 the array has at most limit elements and the last one holds the rest of the input; if limit <= 0 the regex is applied as many times as possible and trailing empty strings are kept (Spark maps limit = 0 to -1 to avoid Java's trailing-empty removal). Available since 1.5.0.

str and regex are implicitly cast to STRING and limit to INT (limit may be a column). Result type is ARRAY<STRING> with containsNull = false (element type follows str's collation in 4.0+). NULL if any argument is NULL. An empty regex is special-cased in UTF8String.split: it splits into individual characters (code points) without the trailing empty element.

Examples
SELECT split('oneAtwoBthreeC', '[ABC]');
-- ["one","two","three",""]
SELECT split('oneAtwoBthreeC', '[ABC]', -1);
-- ["one","two","three",""]
SELECT split('oneAtwoBthreeC', '[ABC]', 2);
-- ["one","twoBthreeC"]
SELECT split('aa1cc2ee3', '[1-9]+', 2);
-- ["aa","cc2ee3"]
SELECT split('hello', '');
-- ["h","e","l","l","o"]
SELECT split('hello', '', 1);
-- ["hello"]
SELECT split('hello', '', 3);
-- ["h","e","llo"]   (4.1+; ["h","e","l"] in 3.4 to 4.0)
SELECT split('', '');
-- [""]
SELECT split('abc', null);
-- NULL
Edge cases to test
  • NULL str, regex or limit, literal and column: NULL.
  • limit of 0, negative, 1, exactly the number of pieces, and larger. split('aacbbcddc', 'c', 0) is ["aa","bb","dd",""] (RegexpExpressionsSuite), same as with -1; split('aa2bb3cc4', '[1-9]+', -2) is ["aa","bb","cc",""] (StringFunctionsSuite).
  • Empty regex: split('hello', '', 5) and split('hello', '', 100) are ["h","e","l","l","o"]; split('', '', n) is [""] for every n. Splitting is per code point (UTF-8 lead byte), so a supplementary character stays whole.
  • Empty str with a non-empty regex: split('', ',') is [""] (Java String.split returns the input when nothing matches).
  • Leading empties follow Java String.split: a positive-width match at the start produces a leading "" (split(',a', ',') is ["","a"]), but a zero-width match at the start does not. split('abc', 'x*') is ["a","b","c",""] in Spark, while the regex crate's split yields a leading "" (["","a","b","c",""]).
  • Empty matches after a non-empty match are reported by Java but not by the regex crate (its testdata/empty.toml: b| on abc gives 3 matches, Java 4), so splits on patterns that can match empty differ.
  • Invalid regex: split does not wrap the error. Spark throws the raw java.util.regex.PatternSyntaxException (no error condition), unlike the other regex functions which use INVALID_PARAMETER_VALUE.PATTERN. Implementations should still raise an error.
  • Column regex and column limit (SPARK-47845 test in StringFunctionsSuite).
  • spark.sql.parser.escapedStringLiterals (default false) only affects parsing of SQL literals before the function. In .slt, DataFusion keeps backslashes literal, so split(s, '\.') splits on a dot.
  • Java vs Rust divergences to decline or test: single-character literal patterns take Java's fast path but match the same as regex; lookaround / backreferences / atomic / possessive (Java only); ASCII \d \w \s in Java vs Unicode in Rust (split('a\u3000b', '\\s'), U+3000 IDEOGRAPHIC SPACE, returns a single element in Spark and two in Rust); . excludes \r, \u0085, \u2028, \u2029 in Java; $ matches before a final line terminator in Java; (?i) ASCII-only in Java; inline flag letters U / u / d; Java-only \Q..\E (commonly used to split on a literal string), \h, \R, \Z; Rust-only (?P<n>), \<, [[:alpha:]].
  • UTF8_LCASE collation (4.0+) prefixes the regex with (?ui).
Differences across Spark versions
  • 4.1.0 (SPARK-49968): with an empty regex and 0 < limit < length, the last element now holds the remainder (split('abcd', '', 2) is ["a","bcd"]). The old behaviour (["a","b"], dropping the rest) is restored by the internal config spark.sql.legacy.truncateForEmptyRegexSplit (default false). 3.4 to 4.0 always truncate.
  • 4.0: collation-aware input types; element type follows str.
Implementation notes
  • Spark: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/regexpExpressions.scala, StringSplit, which calls UTF8String.split / splitLegacyTruncate (common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java) and then java.lang.String.split(regex, limit).
  • DataFusion core has no regex split. string_to_array / split_part split on a literal delimiter. datafusion-spark has neither split nor a regex helper today.
  • Use the shared Java-regex compatibility helper proposed for rlike (verified subset, decline otherwise, decided at planning time for a literal regex). Implement Java String.split semantics directly (leading empty rule, limit handling, keep trailing empties) instead of using Regex::split / splitn. A literal regex with no metacharacters can be split as a plain substring.
  • Belongs in crates/spark/src/function/string/. Add crates/sqllogictest/test_files/string/split.slt and document with #[user_doc].

This issue was generated by an LLM (Claude) from the Apache Spark source code and tests. Examples come from Spark's ExpressionDescription examples, test suites and .sql.out golden files, but please verify them against Spark before relying on them.

Ngôn ngữ chính
Rust
Star
7
Fork
15
Merge trung bình
16 giờ 43 phút
Pull request đã merge (30 ngày)
26

Chuẩn bị môi trường

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của apache/datafusion-spark

Tất cả issue của apache/datafusion-spark

Issue tương tự

Thêm issue về Rust

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.