iconv codec: encoding names that glibc strips to nothing resolve to the locale encoding

Abierto
  • ¿Algo no está bien?
#161,924 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
3/5
Tiempo estimado
1-2 días
Aptitud para principiantes
62/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
python
Área
internationalization

Línea de trabajo

The bug is in iconv_search_function in Lib/encodings/init.py, which probes each name with _iconv_encode(name, '') and accepts any name glibc does not reject. Reproduce with codecs.lookup('!!!') or 'é'.encode('!!!') on Linux under a UTF-8 locale. Done means those names raise LookupError again while valid encodings such as 'utf-8' and 'latin-1' still resolve. The reporter's proposed check, requiring an ASCII alphanumeric before the first '/' or ',', is a starting point to verify.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

3.16 stdlib topic-unicode type-bug
Bug report
Bug description

Since the iconv-based codec (gh-152997, GH-153001) landed, codecs.lookup() on Linux/glibc accepts encoding names whose charset part glibc's iconv_open() reduces to nothing, and returns a codec for the current locale's encoding. Before 3.16 these names raise LookupError.

>>> 'é'.encode('!!!')
b'\xc3\xa9'                    # 3.16.0a2, LC_ALL=C.UTF-8;  3.15: LookupError: unknown encoding: !!!
>>> bytes('é', '한글')
b'\xc3\xa9'
>>> import codecs; codecs.lookup('@').name
'@'

The result depends on the environment: with LC_ALL=C the same call fails with UnicodeEncodeError: '!!!' codec can't encode character '\xe9', because the codec is now ASCII.

name 3.15.2 3.16.0a2 (Linux, UTF-8 locale)
'!!!', '@', '()', 'é', '한글' LookupError codec for the locale encoding
'iconv:한글' LookupError codec for the locale encoding
',', '/', '//TRANSLIT', '//abc' LookupError codec for the locale encoding
'no-such-codec', '8', 'ü8' LookupError LookupError
Cause

iconv_search_function in Lib/encodings/__init__.py rejects only the empty name and then probes the name with _iconv_encode(name, ''). glibc's iconv_open() first removes trailing whitespace, / and , and any //-suffix such as //TRANSLIT (gconv_parse_code() in iconv/gconv_charset.c), then drops every remaining character that is not an ASCII alphanumeric, / or one of _-.,: (strip() in iconv/gconv_charset.h), and treats the resulting empty name as the charset of the current locale (__gconv_open in iconv/gconv_open.c: "If the string is empty define this to mean the charset of the currently selected locale"). So the probe succeeds for any name made only of dropped characters, and also for names whose charset part is empty once the suffix is removed, such as ',', '/', '//TRANSLIT' and '//abc'.

    def iconv_search_function(encoding):
        ...
        if not name:
            return None
        # Test if the encoding is supported by iconv.
        try:
            _iconv_encode(name, '')
        except (LookupError, OSError):
            return None
Possible fix

Before probing, reject a name unless the part before its first / or , contains an ASCII alphanumeric. Requiring an alphanumeric anywhere in the name is not enough: '//TRANSLIT' and '//abc' contain one and still resolve to the locale encoding.

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Lenguaje dominante
Python
Estrellas
78.2k
Forks
36.6k
Merge medio
19 h 57 min
PR fusionados (30 d)
630

Preparar el entorno

Abrir en Codespaces

Inicia el contenedor de desarrollo del proyecto en tu navegador, con tu propia cuenta de GitHub.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de python/cpython

Todos los issues de python/cpython

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.