Implement Spark-compatible `regexp_extract_all`

Aperta
  • Qualcosa non va?
#257 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
22/100
Tipo di issue
Funzionalità
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
rust, spark, sql
Ambito
backend, databases

Direzione di ricerca

Start by reading RegExpExtractAll and RegExpExtractBase.checkGroupIndex in Spark's regexpExpressions.scala, then look at an existing string function under crates/spark/src/function/string/ to copy the registration, #[user_doc] and error-reporting patterns. This depends on the shared Java-regex compatibility helper proposed for rlike, so check its status before writing matching logic; the empty-match iteration rule and group index errors are the parts that must be reproduced exactly. Done means a new function plus crates/sqllogictest/test_files/string/regexp_extract_all.slt covering the NULL, no-match, out-of-range idx and optional-group cases above.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

enhancement

Part of #174 and apache/datafusion-comet#7300.

What the function does

regexp_extract_all(str, regexp[, idx]) returns an array with group idx of every match of the Java regular expression regexp in str. idx defaults to 1; 0 means the whole match. Spark loops Matcher.find() and, for each match, checks 0 <= idx <= groupCount and appends group(idx), or '' when the group did not participate. Available since 3.1.0.

str and regexp are implicitly cast to STRING and idx to INT. idx may be a column. Result type is ARRAY<STRING> (element type follows str's collation in 4.0+, containsNull = true but elements are never NULL). NULL if any argument is NULL. No match returns an empty array.

Examples
SELECT regexp_extract_all('100-200, 300-400', '(\\d+)-(\\d+)', 1);
-- ["100","300"]
SELECT regexp_extract_all('1a 2b 14m', '(\\d+)([a-z]+)', 0);
-- ["1a","2b","14m"]
SELECT regexp_extract_all('1a 2b 14m', '(\\d+)([a-z]+)', 2);
-- ["a","b","m"]
SELECT regexp_extract_all('a 2b 14m', '(\\d+)?([a-z]+)', 1);
-- ["","2","14"]
SELECT regexp_extract_all('1a 2b 14m', '(\\d+)?', 1);
-- ["1","","","2","","","14","",""]
SELECT regexp_extract_all('1a 2b 14m', '\\d+');
-- error INVALID_PARAMETER_VALUE.REGEX_GROUP_INDEX (groupCount 0, groupIndex 1)
SELECT regexp_extract_all('abc', col0, 1) FROM VALUES('], [') AS t(col0);
-- error INVALID_PARAMETER_VALUE.PATTERN
Edge cases to test
  • NULL str, NULL regexp, NULL idx (literal and column): NULL.
  • No match: []. regexp_extract_all('100-200,300-400,500-600', '([a-z])', 1) is [] (RegexpExpressionsSuite).
  • Group index validation only happens inside the match loop, so an out-of-range idx raises no error when there is no match: regexp_extract_all('xyz', '\\d+', 1) returns [].
  • Out-of-range or negative idx with a match: INVALID_PARAMETER_VALUE.REGEX_GROUP_INDEX (SQLSTATE 22023), message The value of parameter(s) `idx` in `regexp_extract_all` is invalid: Expects group index between 0 and 2, but got 3.
  • Optional group that did not participate yields '', not NULL (see the (\\d+)?([a-z]+) example).
  • Empty pattern: matches at every position, so regexp_extract_all('abc', '', 0) returns four empty strings.
  • Empty matches after a non-empty match: the (\\d+)? example above has 9 elements because Java reports the empty match at the end of each \d+ match. The regex crate does not report an empty match that starts where the previous match ended (its testdata/empty.toml gives 3 matches for b| on abc, Java finds 4), so a direct captures_iter port returns fewer elements. Iteration must follow Java's rule.
  • Invalid pattern: INVALID_PARAMETER_VALUE.PATTERN with value the quoted pattern, also for per-row patterns ('] [', '], [').
  • Non-literal pattern column with different patterns per row (Spark caches the last regex).
  • spark.sql.parser.escapedStringLiterals (default false) changes how the SQL literal is unescaped; the kernel sees the result. In .slt, DataFusion keeps backslashes literal ('(\d+)').
  • Java vs Rust pattern divergences to decline or test: lookaround / backreferences / atomic / possessive (Java only; e.g. (?=[^ ]*[0-9])[^ ]+); ASCII \d \w \s in Java vs Unicode in Rust (regexp_extract_all('1\u0663', '\\d', 0) (U+0663 ARABIC-INDIC DIGIT THREE) is ["1"] in Spark); . excludes \r, \u0085, \u2028, \u2029 in Java; $ matches before a final line terminator in Java; (?i) ASCII-only in Java; inline flag letters U / u / d mean different things or are unknown; named groups: Java (?<n>...) only, Rust also (?P<n>...); both number named groups in order, so groupCount matches for accepted patterns.
  • Unicode / supplementary characters in str (Java iterates UTF-16 but never splits a surrogate pair for .).
  • UTF8_LCASE collation (4.0+) compiles with CASE_INSENSITIVE | UNICODE_CASE.
Differences across Spark versions
  • 3.4: group index errors were _LEGACY_ERROR_TEMP_2006 (The specified group index cannot be less than zero.) and _LEGACY_ERROR_TEMP_2007 (Regex group count is <groupCount>, but the specified group index is <groupIndex>.), both SparkIllegalArgumentException. 3.5+ uses INVALID_PARAMETER_VALUE.REGEX_GROUP_INDEX for both cases.
  • 4.0: collation-aware inputs; array element type follows str.
Implementation notes
  • Spark: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/regexpExpressions.scala, RegExpExtractAll (base RegExpExtractBase, RegExpExtractBase.checkGroupIndex).
  • DataFusion core has no equivalent. regexp_match returns the capture groups of the first match only, uses Rust syntax, and returns NULL on no match.
  • Use the shared Java-regex compatibility helper proposed for rlike: accept only a verified subset, return an "unsupported pattern" error otherwise (at planning time for literal patterns), and implement Java's empty-match iteration rule. regexp_count and regexp_substr are rewritten by Spark into this function and regexp_extract, so the same core can back them.
  • Belongs in crates/spark/src/function/string/. Add crates/sqllogictest/test_files/string/regexp_extract_all.slt and document with #[user_doc].

This issue was generated by an LLM (Claude) from the Apache Spark source code and tests. Examples come from Spark's ExpressionDescription examples, test suites and .sql.out golden files, but please verify them against Spark before relying on them.

Lingua principale
Rust
Stelle
7
Fork
15
Merge medio
16h 43m
PR unite (30g)
26

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di apache/datafusion-spark

Tutte le issue di apache/datafusion-spark

Issue simili

Altre issue su Rust

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.