regex: Improper handling of Unicode Properties in character set 3.16a2

Offen
  • Stimmt etwas nicht?
#161,814 1 Kommentar 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen

Maintainer antworten meist innerhalb von 1 Tag

Dieses Issue hat noch niemand übernommen.

Bewertung

Schwierigkeit
4/5
Geschätzter Aufwand
3-5 Tage
Anfängerfreundlichkeit
56/100
Issue-Typ
Bug
Klarheit
Klar beschrieben
Aktivitätsstatus
Aktiv
Tech-Stack
python
Bereich
backend

Rechercherichtung

Start in Lib/re/_compiler.py at _optimize_charset() and inspect how IN_UNI_IGNORE handles category items when a set contains cased literals or ranges. Use the examples in the issue as regression cases, then run the relevant regular expression tests. Done means the category properties and their negations preserve the reported IGNORECASE behavior while existing set matching continues to work.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Beschreibung

type-bug
Bug report

I found this when evaluating the Unicode Properties in the recent alpha release for Re. I realize this work is a WIP, but figured I'd share in case this issue wasn't already known.

With re.IGNORECASE, a character set that contains a case-sensitive property escape (\p{Lu}, \p{Lt}, \p{Uppercase}, \p{Lowercase}, or their \P{...} forms) together with a cased literal or range gives the wrong results. Adding a member to a set can remove matches:

>>> import re
>>> re.fullmatch(r'[\p{Lu}]', 'A', re.IGNORECASE)
<re.Match object; span=(0, 1), match='A'>
>>> re.fullmatch(r'[\p{Lu}x]', 'A', re.IGNORECASE)    # expected a match
>>> re.fullmatch(r'[\p{Lu}x]', 'A')
<re.Match object; span=(0, 1), match='A'>

Below is info gathered by AI, hopefully it is helpful, but the important part is the behavior that doesn't match expectations.

More examples, all with re.IGNORECASE:

Pattern String Result Expected
[\p{Lu}x] 'A' no match match
[\p{Lu}a-c] 'Z' no match match
[^\p{Lu}x] 'A' match no match
[\P{Lu}x] 'A' match no match
[\p{Lt}x] 'Dž' no match match
[\p{Uppercase}x] 'A' no match match
[\p{Lowercase}x] 'A' match no match

"Expected" is how these properties already behave with IGNORECASE outside a set, or as the only member of a
set: \p{Lu} and [\p{Lu}] match 'A' but not 'a'.

The same happens with an inline (?i:...), and with re.ASCII | re.IGNORECASE for ASCII characters. Sets without
a cased literal or range, such as [\p{Lu}1] and [\p{Lu}\d], are not affected.

Cause: in Lib/re/_compiler.py, when a set under IGNORECASE contains a cased literal or range (hascased
from _optimize_charset()), the whole set is compiled as IN_UNI_IGNORE (IN_IGNORE with ASCII). Those
opcodes lowercase the character before testing it against every item in the set, including CATEGORY items. The
literals in the set are already lowercased, so they still match correctly, but the categories are now tested
against the lowercased character: 'A' becomes 'a' before the Lu test.

For the categories that existed before 3.16 (\d, \s, \w), lowercasing a character never changes whether it is
in the category, so this was harmless. It does for the new CATEGORY_LU, CATEGORY_TITLE, CATEGORY_UPPER and
CATEGORY_LOWER, and their negations. Checking every code point that changes when lowercased, these are the only
supported properties whose membership changes (Lu, Lt, Uppercase, Lowercase).

A fix could test CATEGORY items against the original character rather than the lowercased one in the
ignore-case set opcodes, or have the compiler keep those categories out of the ignore-case check.

CPython versions tested on:

3.16

Operating systems tested on:

macOS

Vorherrschende Sprache
Python
Sterne
78.2k
Forks
36.6k
Ø Merge
19 Std. 57 Min.
Gemergte PRs (30 T.)
630

Entwicklungsumgebung

In Codespaces öffnen

Startet den Dev-Container des Projekts im Browser, mit Ihrem eigenen GitHub-Konto.

Erste Schritte

  1. Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
  2. Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
  3. Forken Sie das Repository und arbeiten Sie in einem Branch.
  4. Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.

Mehr aus python/cpython

Alle Issues in python/cpython

Ähnliche Issues

Weitere Issues zu Python

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.