Implement Spark-compatible `regexp_instr`

オープン
  • 何か問題がありますか？
#260 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

まだ誰も着手していません。

評価

難易度
4/5
見積もり時間
3〜5日
初心者へのやさしさ
32/100
issue の種類
機能追加
明瞭さ
おおむね明確
活発さ
活発
技術スタック
java, rust
領域
backend, databases

調査の方向性

Start by reading how an existing string function is registered and documented in crates/spark/src/function/string/, then compare with DataFusion core's datafusion/functions/src/regex/regexpinstr.rs and Spark's RegExpInStr in regexpExpressions.scala. The open question is the shared Java-regex compatibility helper proposed for rlike, since the Rust regex crate cannot be wrapped as-is. Done means regexp_instr is registered with #[user_doc], converts the byte offset to a UTF-16 offset, and crates/sqllogictest/test_files/string/regexp_instr.slt covers the NULL, idx, empty-pattern and invalid-pattern cases above.

索引モデルが issue の本文から書いたものです。

説明

enhancement

Part of #174 and apache/datafusion-comet#7300.

What the function does

regexp_instr(str, regexp[, idx]) returns the 1-based position where the first match of the Java regular expression regexp begins in str, or 0 if there is no match. Spark computes Matcher.find() then start() + 1. Available since 3.4.0.

str and regexp are implicitly cast to STRING and the optional third argument idx to INT (default 0). idx is accepted but ignored by the evaluation, although a NULL idx still makes the result NULL. Result is INT. NULL if any argument is NULL.

Examples
SELECT regexp_instr(r"\abc", r"^\\abc$");
-- 1
SELECT regexp_instr('[email protected]', '@[^.]*');
-- 5
SELECT regexp_instr('abc', 'b');
-- 2
SELECT regexp_instr('abc', 'x');
-- 0
SELECT regexp_instr('ABC', '(?-i)b');
-- 0
SELECT regexp_instr('1a 2b 14m', '\\d{2}(a|b|m)');
-- 7
SELECT regexp_instr('abc', null);
-- NULL
SELECT regexp_instr('abc', col0, 1) FROM VALUES(') ?') AS t(col0);
-- error INVALID_PARAMETER_VALUE.PATTERN
Edge cases to test
  • NULL str, NULL regexp, NULL idx (literal and column): NULL.
  • idx values do not change the result: regexp_instr('abc', 'b', 5) and regexp_instr('abc', 'b', -1) both return 2 (no group-index check is performed).
  • Empty pattern: matches at position 0, so the result is 1 (also for str = '').
  • The position is Matcher.start() + 1, a UTF-16 code unit index, not a code point index. regexp_instr('😀a', 'a') returns 3 in Spark (the emoji is a surrogate pair). DataFusion's regexp_instr counts characters and returns 2; a byte-offset port would return 5.
  • Invalid pattern: INVALID_PARAMETER_VALUE.PATTERN, message The value of parameter(s) `regexp` in `regexp_instr` is invalid: ', ('. (RegexpExpressionsSuite), for literal and per-row patterns.
  • Non-literal pattern columns that change per row.
  • spark.sql.parser.escapedStringLiterals (default false): the first example uses raw literals; with the config on, '\abc' and '^\\abc$' reach the function unchanged. In .slt, DataFusion keeps backslashes literal.
  • Java vs Rust divergences to decline or test: lookaround / backreferences / atomic / possessive (Java only); ASCII \d \w \s in Java vs Unicode in Rust (regexp_instr('\u06631', '\\d'), U+0663 ARABIC-INDIC DIGIT THREE, is 2 in Spark); . excludes \r, \u0085, \u2028, \u2029 in Java; $ matches before a final line terminator in Java (regexp_instr('ab\n', 'b$') is 2); (?i) ASCII-only in Java; inline flag letters (U, u, d) and the (?-i) example; Java-only \Q..\E, \h, \R, \Z, octal; Rust-only (?P<n>), \<, [[:alpha:]].
  • UTF8_LCASE collation (4.0+) matches case-insensitively.
Differences across Spark versions
  • 4.0: collation-aware input types. Otherwise none found (3.4.0 through 4.2.0).
Implementation notes
  • Spark: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/regexpExpressions.scala, RegExpInStr (extends RegExpExtractBase; catches IllegalStateException and returns 0).
  • DataFusion core regexp_instr(str, regexp[, start[, N[, flags[, subexpr]]]]) (datafusion/functions/src/regex/regexpinstr.rs) returns BIGINT, treats the third argument as a start position, counts code points, and uses Rust regex syntax. It cannot be wrapped as-is.
  • Use the shared Java-regex compatibility helper proposed for rlike (verified subset, decline otherwise, decided at planning time for literal patterns). Convert the Rust byte offset to a UTF-16 offset (count code points, adding 1 extra for each code point above U+FFFF) before adding 1.
  • Belongs in crates/spark/src/function/string/. Add crates/sqllogictest/test_files/string/regexp_instr.slt and document with #[user_doc].

This issue was generated by an LLM (Claude) from the Apache Spark source code and tests. Examples come from Spark's ExpressionDescription examples, test suites and .sql.out golden files, but please verify them against Spark before relying on them.

主要言語
Rust
スター
7
フォーク
15
平均マージ
16時間 43分
マージ済み PR（30日）
26

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

apache/datafusion-spark のほかの issue

apache/datafusion-spark の issue をすべて見る

似ている issue

Rust の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。