regex: Improper handling of Unicode Properties in character set 3.16a2

Ouverte
  • Quelque chose ne va pas ?
#161,814 1 commentaire 0 réactions 0 personnes assignées Voir sur GitHub

Les mainteneurs répondent en général sous 1 jour

Personne n'a encore pris cette issue.

Évaluation

Difficulté
4/5
Temps estimé
3-5 jours
Accessibilité débutants
56/100
Type d'issue
Bug
Clarté
Clairement spécifiée
Activité
Active
Stack technique
python
Domaine
backend

Piste de recherche

Start in Lib/re/_compiler.py at _optimize_charset() and inspect how IN_UNI_IGNORE handles category items when a set contains cased literals or ranges. Use the examples in the issue as regression cases, then run the relevant regular expression tests. Done means the category properties and their negations preserve the reported IGNORECASE behavior while existing set matching continues to work.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Description

type-bug
Bug report

I found this when evaluating the Unicode Properties in the recent alpha release for Re. I realize this work is a WIP, but figured I'd share in case this issue wasn't already known.

With re.IGNORECASE, a character set that contains a case-sensitive property escape (\p{Lu}, \p{Lt}, \p{Uppercase}, \p{Lowercase}, or their \P{...} forms) together with a cased literal or range gives the wrong results. Adding a member to a set can remove matches:

>>> import re
>>> re.fullmatch(r'[\p{Lu}]', 'A', re.IGNORECASE)
<re.Match object; span=(0, 1), match='A'>
>>> re.fullmatch(r'[\p{Lu}x]', 'A', re.IGNORECASE)    # expected a match
>>> re.fullmatch(r'[\p{Lu}x]', 'A')
<re.Match object; span=(0, 1), match='A'>

Below is info gathered by AI, hopefully it is helpful, but the important part is the behavior that doesn't match expectations.

More examples, all with re.IGNORECASE:

Pattern String Result Expected
[\p{Lu}x] 'A' no match match
[\p{Lu}a-c] 'Z' no match match
[^\p{Lu}x] 'A' match no match
[\P{Lu}x] 'A' match no match
[\p{Lt}x] 'Dž' no match match
[\p{Uppercase}x] 'A' no match match
[\p{Lowercase}x] 'A' match no match

"Expected" is how these properties already behave with IGNORECASE outside a set, or as the only member of a
set: \p{Lu} and [\p{Lu}] match 'A' but not 'a'.

The same happens with an inline (?i:...), and with re.ASCII | re.IGNORECASE for ASCII characters. Sets without
a cased literal or range, such as [\p{Lu}1] and [\p{Lu}\d], are not affected.

Cause: in Lib/re/_compiler.py, when a set under IGNORECASE contains a cased literal or range (hascased
from _optimize_charset()), the whole set is compiled as IN_UNI_IGNORE (IN_IGNORE with ASCII). Those
opcodes lowercase the character before testing it against every item in the set, including CATEGORY items. The
literals in the set are already lowercased, so they still match correctly, but the categories are now tested
against the lowercased character: 'A' becomes 'a' before the Lu test.

For the categories that existed before 3.16 (\d, \s, \w), lowercasing a character never changes whether it is
in the category, so this was harmless. It does for the new CATEGORY_LU, CATEGORY_TITLE, CATEGORY_UPPER and
CATEGORY_LOWER, and their negations. Checking every code point that changes when lowercased, these are the only
supported properties whose membership changes (Lu, Lt, Uppercase, Lowercase).

A fix could test CATEGORY items against the original character rather than the lowercased one in the
ignore-case set opcodes, or have the compiler keep those categories out of the ignore-case check.

CPython versions tested on:

3.16

Operating systems tested on:

macOS

Langage dominant
Python
Étoiles
78.2k
Forks
36.6k
Merge moyen
19 h 57 min
PR mergées (30 j)
630

Préparer son environnement

Ouvrir dans Codespaces

Lance le conteneur de développement du projet dans votre navigateur, avec votre propre compte GitHub.

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Autres issues de python/cpython

Toutes les issues de python/cpython

Issues similaires

Plus d'issues Python

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.