Support DBD::SQLite FTS4 perl custom tokenizers

Open
#1,459 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
44/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
java, perl, sqlite
Domain
backend, databases

Research direction

Start with t/Search/Fulltext/Tokenizer/Ngram.t and reproduce the Unigram tokenizer failure on both execution backends. Trace Search::Fulltext::SQLite through CREATE VIRTUAL TABLE handling in the bundled DBI/SQLite implementation, then add the focused FTS4 callback test described in the issue. Done means the perl tokenizer syntax works and the integration test passes on both backends.

Written by the indexing model from the issue text.

Description

area:cpan-port bug

Summary

PerlOnJava's bundled JDBC-backed DBI/SQLite implementation does not support the perl custom tokenizer used by DBD::SQLite FTS4. This prevents Search::Fulltext from using pure-Perl custom tokenizers.

Reproduction

Distribution: Search-Fulltext-Tokenizer-Ngram-0.01

The upstream test constructs Search::Fulltext with:

tokenizer => "perl 'Search::Fulltext::Tokenizer::${tokenizer_name}::get_tokenizer'"

The equivalent minimal reproducer is:

use Search::Fulltext;
Search::Fulltext->new(
    docs      => ['abc'],
    tokenizer => q(perl "Search::Fulltext::Tokenizer::Bigram::get_tokenizer"),
);

PerlOnJava reports:

DBI execute failed: unknown tokenizer: perl

Results

  • Archived CPAN run: 20260918-141920-96054
  • Standard Perl: PASS, 4 tests
  • PerlOnJava JVM backend: FAIL
  • PerlOnJava interpreter backend: FAIL
  • Failing test: t/Search/Fulltext/Tokenizer/Ngram.t, Unigram tokenizer subtest
  • The failure occurs when Search::Fulltext::SQLite executes CREATE VIRTUAL TABLE ... USING fts4(..., tokenize=perl ...).

The target and dependency distributions contain no XS/native code. This is a bundled DBI/SQLite compatibility gap, not an upstream test or environment failure.

Expected behavior

Support the perl FTS4 tokenizer syntax used by DBD::SQLite, or provide a documented compatible implementation that invokes the requested Perl tokenizer callback.

Suggested regression coverage

Add a focused DBI/SQLite test that creates an FTS4 table with tokenize=perl and exercises a Perl tokenizer callback, then retain Search::Fulltext::Tokenizer::Ngram as integration coverage on both execution backends.

Dominant language
Perl
Stars
64
Forks
6
Avg merge
5h 25m
Merged PRs (30d)
157

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from fglock/PerlOnJava

All issues in fglock/PerlOnJava

Similar issues

More Perl issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.