regex: Improper handling of Unicode Properties in character set 3.16a2

Abierto
  • ¿Algo no está bien?
#161,814 1 comentario 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
56/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
python
Área
backend

Línea de trabajo

Start in Lib/re/_compiler.py at _optimize_charset() and inspect how IN_UNI_IGNORE handles category items when a set contains cased literals or ranges. Use the examples in the issue as regression cases, then run the relevant regular expression tests. Done means the category properties and their negations preserve the reported IGNORECASE behavior while existing set matching continues to work.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

type-bug
Bug report

I found this when evaluating the Unicode Properties in the recent alpha release for Re. I realize this work is a WIP, but figured I'd share in case this issue wasn't already known.

With re.IGNORECASE, a character set that contains a case-sensitive property escape (\p{Lu}, \p{Lt}, \p{Uppercase}, \p{Lowercase}, or their \P{...} forms) together with a cased literal or range gives the wrong results. Adding a member to a set can remove matches:

>>> import re
>>> re.fullmatch(r'[\p{Lu}]', 'A', re.IGNORECASE)
<re.Match object; span=(0, 1), match='A'>
>>> re.fullmatch(r'[\p{Lu}x]', 'A', re.IGNORECASE)    # expected a match
>>> re.fullmatch(r'[\p{Lu}x]', 'A')
<re.Match object; span=(0, 1), match='A'>

Below is info gathered by AI, hopefully it is helpful, but the important part is the behavior that doesn't match expectations.

More examples, all with re.IGNORECASE:

Pattern String Result Expected
[\p{Lu}x] 'A' no match match
[\p{Lu}a-c] 'Z' no match match
[^\p{Lu}x] 'A' match no match
[\P{Lu}x] 'A' match no match
[\p{Lt}x] 'Dž' no match match
[\p{Uppercase}x] 'A' no match match
[\p{Lowercase}x] 'A' match no match

"Expected" is how these properties already behave with IGNORECASE outside a set, or as the only member of a
set: \p{Lu} and [\p{Lu}] match 'A' but not 'a'.

The same happens with an inline (?i:...), and with re.ASCII | re.IGNORECASE for ASCII characters. Sets without
a cased literal or range, such as [\p{Lu}1] and [\p{Lu}\d], are not affected.

Cause: in Lib/re/_compiler.py, when a set under IGNORECASE contains a cased literal or range (hascased
from _optimize_charset()), the whole set is compiled as IN_UNI_IGNORE (IN_IGNORE with ASCII). Those
opcodes lowercase the character before testing it against every item in the set, including CATEGORY items. The
literals in the set are already lowercased, so they still match correctly, but the categories are now tested
against the lowercased character: 'A' becomes 'a' before the Lu test.

For the categories that existed before 3.16 (\d, \s, \w), lowercasing a character never changes whether it is
in the category, so this was harmless. It does for the new CATEGORY_LU, CATEGORY_TITLE, CATEGORY_UPPER and
CATEGORY_LOWER, and their negations. Checking every code point that changes when lowercased, these are the only
supported properties whose membership changes (Lu, Lt, Uppercase, Lowercase).

A fix could test CATEGORY items against the original character rather than the lowercased one in the
ignore-case set opcodes, or have the compiler keep those categories out of the ignore-case check.

CPython versions tested on:

3.16

Operating systems tested on:

macOS

Lenguaje dominante
Python
Estrellas
78.2k
Forks
36.6k
Merge medio
19 h 57 min
PR fusionados (30 d)
630

Preparar el entorno

Abrir en Codespaces

Inicia el contenedor de desarrollo del proyecto en tu navegador, con tu propia cuenta de GitHub.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de python/cpython

Todos los issues de python/cpython

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.