iconv codec: encoding names that glibc strips to nothing resolve to the locale encoding

Ouverte
  • Quelque chose ne va pas ?
#161,924 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub

Les mainteneurs répondent en général sous 1 jour

Personne n'a encore pris cette issue.

Évaluation

Difficulté
3/5
Temps estimé
1-2 jours
Accessibilité débutants
62/100
Type d'issue
Bug
Clarté
Clairement spécifiée
Activité
Active
Stack technique
python
Domaine
internationalization

Piste de recherche

The bug is in iconv_search_function in Lib/encodings/init.py, which probes each name with _iconv_encode(name, '') and accepts any name glibc does not reject. Reproduce with codecs.lookup('!!!') or 'é'.encode('!!!') on Linux under a UTF-8 locale. Done means those names raise LookupError again while valid encodings such as 'utf-8' and 'latin-1' still resolve. The reporter's proposed check, requiring an ASCII alphanumeric before the first '/' or ',', is a starting point to verify.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Description

3.16 stdlib topic-unicode type-bug
Bug report
Bug description

Since the iconv-based codec (gh-152997, GH-153001) landed, codecs.lookup() on Linux/glibc accepts encoding names whose charset part glibc's iconv_open() reduces to nothing, and returns a codec for the current locale's encoding. Before 3.16 these names raise LookupError.

>>> 'é'.encode('!!!')
b'\xc3\xa9'                    # 3.16.0a2, LC_ALL=C.UTF-8;  3.15: LookupError: unknown encoding: !!!
>>> bytes('é', '한글')
b'\xc3\xa9'
>>> import codecs; codecs.lookup('@').name
'@'

The result depends on the environment: with LC_ALL=C the same call fails with UnicodeEncodeError: '!!!' codec can't encode character '\xe9', because the codec is now ASCII.

name 3.15.2 3.16.0a2 (Linux, UTF-8 locale)
'!!!', '@', '()', 'é', '한글' LookupError codec for the locale encoding
'iconv:한글' LookupError codec for the locale encoding
',', '/', '//TRANSLIT', '//abc' LookupError codec for the locale encoding
'no-such-codec', '8', 'ü8' LookupError LookupError
Cause

iconv_search_function in Lib/encodings/__init__.py rejects only the empty name and then probes the name with _iconv_encode(name, ''). glibc's iconv_open() first removes trailing whitespace, / and , and any //-suffix such as //TRANSLIT (gconv_parse_code() in iconv/gconv_charset.c), then drops every remaining character that is not an ASCII alphanumeric, / or one of _-.,: (strip() in iconv/gconv_charset.h), and treats the resulting empty name as the charset of the current locale (__gconv_open in iconv/gconv_open.c: "If the string is empty define this to mean the charset of the currently selected locale"). So the probe succeeds for any name made only of dropped characters, and also for names whose charset part is empty once the suffix is removed, such as ',', '/', '//TRANSLIT' and '//abc'.

    def iconv_search_function(encoding):
        ...
        if not name:
            return None
        # Test if the encoding is supported by iconv.
        try:
            _iconv_encode(name, '')
        except (LookupError, OSError):
            return None
Possible fix

Before probing, reject a name unless the part before its first / or , contains an ASCII alphanumeric. Requiring an alphanumeric anywhere in the name is not enough: '//TRANSLIT' and '//abc' contain one and still resolve to the locale encoding.

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Langage dominant
Python
Étoiles
78.2k
Forks
36.6k
Merge moyen
19 h 57 min
PR mergées (30 j)
630

Préparer son environnement

Ouvrir dans Codespaces

Lance le conteneur de développement du projet dans votre navigateur, avec votre propre compte GitHub.

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Autres issues de python/cpython

Toutes les issues de python/cpython

Issues similaires

Plus d'issues Python

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.