Fix \\G matching for interpolated /x patterns used by String::Formatter

Open
#1,364 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
java, perl
Domain
backend, compilers

Research direction

Start with the regex-engine and Joni integration paths, tracing the interpolated qr// pattern through compilation and matching on both JVM and interpreter backends. Compare the minimal String::Formatter-style pattern with its non-interpolated, non-/x, and non-\G variants, recording captures and pos values. Done means a project-owned regression test passes unchanged under system Perl and both PerlOnJava backends, followed by the String::Errf and adjacent regex compatibility tests.

Written by the indexing model from the issue text.

Description

area:backend area:cpan-port area:regex bug

Summary

Fix a PerlOnJava regex-engine mismatch that prevents String::Formatter from
matching its \G-anchored, interpolated format-token pattern. This causes
String::Errf to return format strings literally instead of expanding them.

CPAN failure

CPAN compatibility run 20260910-101908-8428:

  • Distribution: String-Errf v0.009
  • Target module: String::Errf
  • Result: 3 of 6 subtests failed; 1 of 3 test programs failed
  • Failing test: t/basic.t
  • Dependency: String::Formatter v1.235, pure Perl and successfully built

The failures cover integer, float, timestamp, numbered, string, and undef
formatting paths. The common symptom is that tokens such as %{x}i remain
unchanged instead of being expanded.

Minimal reproducer

The upstream formatter eventually uses this pattern structure:

my $regex = qr/
 (%
  (?:{
    ([^;]*?)
    (?: ; ([^\}]*?) )?
  })
  ($|.)
 )
/xi;

my $s = '%{x}i';
while ($s =~ m{\G(.*?)$regex}gs) {
    print "matched=[$1][$2][$3][$4][$5]\n";
}

Expected under system Perl:

matched=[][%{x}i][x][][i]

Observed under PerlOnJava:

(no match)

The smallest user-visible reproducer is:

use String::Errf errf => ();
print errf('%{x}i', { x => 10 }), "\n";

Expected output:

10

PerlOnJava output:

%{x}i

The same result occurs on the JVM and interpreter backends.

System-Perl comparison

The complete String::Errf upstream test suite passes under system Perl. The
only non-ok output is the distribution's existing TODO-marked local-time
expectation, caused by the test host timezone; the test harness exits
successfully.

The pure-Perl String::Formatter dependency is therefore not the cause. Its
formatter works under system Perl and fails under PerlOnJava at the regex
matching stage.

Ownership and root-cause hypothesis

Primary ownership is the regex engine, specifically the interaction among:

  • a compiled qr// object interpolated into a larger pattern;
  • /x whitespace/comments and nested captures;
  • \G as the independent search-position anchor;
  • /g execution and zero-width/progression state.

This is not a String::Errf or String::Formatter implementation defect.
The existing \G control check with a simple literal pattern succeeds, so the
investigation should preserve the distinction between basic \G support and
the failing interpolated/capture pattern path.

Related but distinct: #1265 concerns unescaped literal-brace handling and
warning publication in a substitution. It may share regex compiler or matcher
infrastructure, but this issue must independently preserve the format-token
match behavior.

Investigation requirements

Trace the first divergence through these stages:

  1. Confirm raw and cooked pattern source, including the interpolated qr//
    identity and /x modifier.
  2. Confirm compiled Joni options, capture numbering, and any optimizer/start
    metadata.
  3. Record search start, region bounds, \G position, and every /g result.
  4. Compare the same pattern with and without interpolation, /x, captures, and
    \G to isolate the smallest failing feature combination.
  5. Test JVM and interpreter paths and direct Joni behavior where applicable.

Do not fix this with a String::Formatter-specific source rewrite, a
java.util.regex fallback, or a special case recognizing %{x}i. Preserve
general Perl matching semantics.

Requested fix

  • Make the failing pattern match with system-Perl-compatible captures and
    progression.
  • Preserve correct \G and /g behavior for adjacent and repeated tokens.
  • Verify interpolated qr// patterns under /x, including empty-prefix and
    non-empty-prefix cases.
  • Retain correct behavior for escaped braces and malformed format patterns.
  • Keep JVM and interpreter results identical.

Regression coverage

Add a tracked project-owned regression test independent of the downloaded CPAN
build tree. It must:

  • pass unchanged under system Perl;
  • fail on the unfixed PerlOnJava parent for the expected mismatch;
  • run on JVM and interpreter after the fix;
  • include the String::Formatter-style minimal pattern;
  • include direct String::Errf formatting for strings, integers, floats,
    numbered values, timestamps, and undef handlers;
  • include a nearby escaped-brace control;
  • record the complete /g match sequence and pos values;
  • cover both interpolated and non-interpolated forms.

After the focused fix, rerun String::Errf v0.009, adjacent regex tests, and
the related String::Formatter compatibility paths. Scan for regressions in
captures, \G, /g, /x, and qr// interpolation.

Labels: bug, area:regex, area:cpan-port, area:backend.

Dominant language
Perl
Stars
64
Forks
6
Avg merge
5h 18m
Merged PRs (30d)
162

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from fglock/PerlOnJava

All issues in fglock/PerlOnJava

Similar issues

More Perl issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.