Implement Spark-compatible `regexp_extract_all`

Abierto
  • ¿Algo no está bien?
#257 1 comentario 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
22/100
Tipo de issue
Nueva funcionalidad
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
rust, spark, sql
Área
backend, databases

Línea de trabajo

Start by reading RegExpExtractAll and RegExpExtractBase.checkGroupIndex in Spark's regexpExpressions.scala, then look at an existing string function under crates/spark/src/function/string/ to copy the registration, #[user_doc] and error-reporting patterns. This depends on the shared Java-regex compatibility helper proposed for rlike, so check its status before writing matching logic; the empty-match iteration rule and group index errors are the parts that must be reproduced exactly. Done means a new function plus crates/sqllogictest/test_files/string/regexp_extract_all.slt covering the NULL, no-match, out-of-range idx and optional-group cases above.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

enhancement

Part of #174 and apache/datafusion-comet#7300.

What the function does

regexp_extract_all(str, regexp[, idx]) returns an array with group idx of every match of the Java regular expression regexp in str. idx defaults to 1; 0 means the whole match. Spark loops Matcher.find() and, for each match, checks 0 <= idx <= groupCount and appends group(idx), or '' when the group did not participate. Available since 3.1.0.

str and regexp are implicitly cast to STRING and idx to INT. idx may be a column. Result type is ARRAY<STRING> (element type follows str's collation in 4.0+, containsNull = true but elements are never NULL). NULL if any argument is NULL. No match returns an empty array.

Examples
SELECT regexp_extract_all('100-200, 300-400', '(\\d+)-(\\d+)', 1);
-- ["100","300"]
SELECT regexp_extract_all('1a 2b 14m', '(\\d+)([a-z]+)', 0);
-- ["1a","2b","14m"]
SELECT regexp_extract_all('1a 2b 14m', '(\\d+)([a-z]+)', 2);
-- ["a","b","m"]
SELECT regexp_extract_all('a 2b 14m', '(\\d+)?([a-z]+)', 1);
-- ["","2","14"]
SELECT regexp_extract_all('1a 2b 14m', '(\\d+)?', 1);
-- ["1","","","2","","","14","",""]
SELECT regexp_extract_all('1a 2b 14m', '\\d+');
-- error INVALID_PARAMETER_VALUE.REGEX_GROUP_INDEX (groupCount 0, groupIndex 1)
SELECT regexp_extract_all('abc', col0, 1) FROM VALUES('], [') AS t(col0);
-- error INVALID_PARAMETER_VALUE.PATTERN
Edge cases to test
  • NULL str, NULL regexp, NULL idx (literal and column): NULL.
  • No match: []. regexp_extract_all('100-200,300-400,500-600', '([a-z])', 1) is [] (RegexpExpressionsSuite).
  • Group index validation only happens inside the match loop, so an out-of-range idx raises no error when there is no match: regexp_extract_all('xyz', '\\d+', 1) returns [].
  • Out-of-range or negative idx with a match: INVALID_PARAMETER_VALUE.REGEX_GROUP_INDEX (SQLSTATE 22023), message The value of parameter(s) `idx` in `regexp_extract_all` is invalid: Expects group index between 0 and 2, but got 3.
  • Optional group that did not participate yields '', not NULL (see the (\\d+)?([a-z]+) example).
  • Empty pattern: matches at every position, so regexp_extract_all('abc', '', 0) returns four empty strings.
  • Empty matches after a non-empty match: the (\\d+)? example above has 9 elements because Java reports the empty match at the end of each \d+ match. The regex crate does not report an empty match that starts where the previous match ended (its testdata/empty.toml gives 3 matches for b| on abc, Java finds 4), so a direct captures_iter port returns fewer elements. Iteration must follow Java's rule.
  • Invalid pattern: INVALID_PARAMETER_VALUE.PATTERN with value the quoted pattern, also for per-row patterns ('] [', '], [').
  • Non-literal pattern column with different patterns per row (Spark caches the last regex).
  • spark.sql.parser.escapedStringLiterals (default false) changes how the SQL literal is unescaped; the kernel sees the result. In .slt, DataFusion keeps backslashes literal ('(\d+)').
  • Java vs Rust pattern divergences to decline or test: lookaround / backreferences / atomic / possessive (Java only; e.g. (?=[^ ]*[0-9])[^ ]+); ASCII \d \w \s in Java vs Unicode in Rust (regexp_extract_all('1\u0663', '\\d', 0) (U+0663 ARABIC-INDIC DIGIT THREE) is ["1"] in Spark); . excludes \r, \u0085, \u2028, \u2029 in Java; $ matches before a final line terminator in Java; (?i) ASCII-only in Java; inline flag letters U / u / d mean different things or are unknown; named groups: Java (?<n>...) only, Rust also (?P<n>...); both number named groups in order, so groupCount matches for accepted patterns.
  • Unicode / supplementary characters in str (Java iterates UTF-16 but never splits a surrogate pair for .).
  • UTF8_LCASE collation (4.0+) compiles with CASE_INSENSITIVE | UNICODE_CASE.
Differences across Spark versions
  • 3.4: group index errors were _LEGACY_ERROR_TEMP_2006 (The specified group index cannot be less than zero.) and _LEGACY_ERROR_TEMP_2007 (Regex group count is <groupCount>, but the specified group index is <groupIndex>.), both SparkIllegalArgumentException. 3.5+ uses INVALID_PARAMETER_VALUE.REGEX_GROUP_INDEX for both cases.
  • 4.0: collation-aware inputs; array element type follows str.
Implementation notes
  • Spark: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/regexpExpressions.scala, RegExpExtractAll (base RegExpExtractBase, RegExpExtractBase.checkGroupIndex).
  • DataFusion core has no equivalent. regexp_match returns the capture groups of the first match only, uses Rust syntax, and returns NULL on no match.
  • Use the shared Java-regex compatibility helper proposed for rlike: accept only a verified subset, return an "unsupported pattern" error otherwise (at planning time for literal patterns), and implement Java's empty-match iteration rule. regexp_count and regexp_substr are rewritten by Spark into this function and regexp_extract, so the same core can back them.
  • Belongs in crates/spark/src/function/string/. Add crates/sqllogictest/test_files/string/regexp_extract_all.slt and document with #[user_doc].

This issue was generated by an LLM (Claude) from the Apache Spark source code and tests. Examples come from Spark's ExpressionDescription examples, test suites and .sql.out golden files, but please verify them against Spark before relying on them.

Lenguaje dominante
Rust
Estrellas
7
Forks
15
Merge medio
16 h 43 min
PR fusionados (30 d)
26

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de apache/datafusion-spark

Todos los issues de apache/datafusion-spark

Issues similares

Más issues de Rust

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.