Implement Spark-compatible `regexp_instr`

Aperta
  • Qualcosa non va?
#260 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
32/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
java, rust
Ambito
backend, databases

Direzione di ricerca

Start by reading how an existing string function is registered and documented in crates/spark/src/function/string/, then compare with DataFusion core's datafusion/functions/src/regex/regexpinstr.rs and Spark's RegExpInStr in regexpExpressions.scala. The open question is the shared Java-regex compatibility helper proposed for rlike, since the Rust regex crate cannot be wrapped as-is. Done means regexp_instr is registered with #[user_doc], converts the byte offset to a UTF-16 offset, and crates/sqllogictest/test_files/string/regexp_instr.slt covers the NULL, idx, empty-pattern and invalid-pattern cases above.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

enhancement

Part of #174 and apache/datafusion-comet#7300.

What the function does

regexp_instr(str, regexp[, idx]) returns the 1-based position where the first match of the Java regular expression regexp begins in str, or 0 if there is no match. Spark computes Matcher.find() then start() + 1. Available since 3.4.0.

str and regexp are implicitly cast to STRING and the optional third argument idx to INT (default 0). idx is accepted but ignored by the evaluation, although a NULL idx still makes the result NULL. Result is INT. NULL if any argument is NULL.

Examples
SELECT regexp_instr(r"\abc", r"^\\abc$");
-- 1
SELECT regexp_instr('[email protected]', '@[^.]*');
-- 5
SELECT regexp_instr('abc', 'b');
-- 2
SELECT regexp_instr('abc', 'x');
-- 0
SELECT regexp_instr('ABC', '(?-i)b');
-- 0
SELECT regexp_instr('1a 2b 14m', '\\d{2}(a|b|m)');
-- 7
SELECT regexp_instr('abc', null);
-- NULL
SELECT regexp_instr('abc', col0, 1) FROM VALUES(') ?') AS t(col0);
-- error INVALID_PARAMETER_VALUE.PATTERN
Edge cases to test
  • NULL str, NULL regexp, NULL idx (literal and column): NULL.
  • idx values do not change the result: regexp_instr('abc', 'b', 5) and regexp_instr('abc', 'b', -1) both return 2 (no group-index check is performed).
  • Empty pattern: matches at position 0, so the result is 1 (also for str = '').
  • The position is Matcher.start() + 1, a UTF-16 code unit index, not a code point index. regexp_instr('😀a', 'a') returns 3 in Spark (the emoji is a surrogate pair). DataFusion's regexp_instr counts characters and returns 2; a byte-offset port would return 5.
  • Invalid pattern: INVALID_PARAMETER_VALUE.PATTERN, message The value of parameter(s) `regexp` in `regexp_instr` is invalid: ', ('. (RegexpExpressionsSuite), for literal and per-row patterns.
  • Non-literal pattern columns that change per row.
  • spark.sql.parser.escapedStringLiterals (default false): the first example uses raw literals; with the config on, '\abc' and '^\\abc$' reach the function unchanged. In .slt, DataFusion keeps backslashes literal.
  • Java vs Rust divergences to decline or test: lookaround / backreferences / atomic / possessive (Java only); ASCII \d \w \s in Java vs Unicode in Rust (regexp_instr('\u06631', '\\d'), U+0663 ARABIC-INDIC DIGIT THREE, is 2 in Spark); . excludes \r, \u0085, \u2028, \u2029 in Java; $ matches before a final line terminator in Java (regexp_instr('ab\n', 'b$') is 2); (?i) ASCII-only in Java; inline flag letters (U, u, d) and the (?-i) example; Java-only \Q..\E, \h, \R, \Z, octal; Rust-only (?P<n>), \<, [[:alpha:]].
  • UTF8_LCASE collation (4.0+) matches case-insensitively.
Differences across Spark versions
  • 4.0: collation-aware input types. Otherwise none found (3.4.0 through 4.2.0).
Implementation notes
  • Spark: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/regexpExpressions.scala, RegExpInStr (extends RegExpExtractBase; catches IllegalStateException and returns 0).
  • DataFusion core regexp_instr(str, regexp[, start[, N[, flags[, subexpr]]]]) (datafusion/functions/src/regex/regexpinstr.rs) returns BIGINT, treats the third argument as a start position, counts code points, and uses Rust regex syntax. It cannot be wrapped as-is.
  • Use the shared Java-regex compatibility helper proposed for rlike (verified subset, decline otherwise, decided at planning time for literal patterns). Convert the Rust byte offset to a UTF-16 offset (count code points, adding 1 extra for each code point above U+FFFF) before adding 1.
  • Belongs in crates/spark/src/function/string/. Add crates/sqllogictest/test_files/string/regexp_instr.slt and document with #[user_doc].

This issue was generated by an LLM (Claude) from the Apache Spark source code and tests. Examples come from Spark's ExpressionDescription examples, test suites and .sql.out golden files, but please verify them against Spark before relying on them.

Lingua principale
Rust
Stelle
7
Fork
15
Merge medio
16h 43m
PR unite (30g)
26

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di apache/datafusion-spark

Tutte le issue di apache/datafusion-spark

Issue simili

Altre issue su Rust

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.