iconv codec: encoding names that glibc strips to nothing resolve to the locale encoding

Aberta
  • Algo errado?
#161,924 0 comentários 0 reações 0 responsáveis Ver no GitHub

Mantenedores costumam responder em até 1 dia

Ninguém assumiu esta issue ainda.

Avaliação

Dificuldade
3/5
Tempo estimado
1-2 dias
Facilidade para iniciantes
62/100
Tipo de issue
Bug
Clareza
Claramente especificada
Status de atividade
Ativa
Stack de tecnologia
python
Domínio
internationalization

Direção de pesquisa

The bug is in iconv_search_function in Lib/encodings/init.py, which probes each name with _iconv_encode(name, '') and accepts any name glibc does not reject. Reproduce with codecs.lookup('!!!') or 'é'.encode('!!!') on Linux under a UTF-8 locale. Done means those names raise LookupError again while valid encodings such as 'utf-8' and 'latin-1' still resolve. The reporter's proposed check, requiring an ASCII alphanumeric before the first '/' or ',', is a starting point to verify.

Escrita pelo modelo de indexação a partir do texto da issue.

Descrição

3.16 stdlib topic-unicode type-bug
Bug report
Bug description

Since the iconv-based codec (gh-152997, GH-153001) landed, codecs.lookup() on Linux/glibc accepts encoding names whose charset part glibc's iconv_open() reduces to nothing, and returns a codec for the current locale's encoding. Before 3.16 these names raise LookupError.

>>> 'é'.encode('!!!')
b'\xc3\xa9'                    # 3.16.0a2, LC_ALL=C.UTF-8;  3.15: LookupError: unknown encoding: !!!
>>> bytes('é', '한글')
b'\xc3\xa9'
>>> import codecs; codecs.lookup('@').name
'@'

The result depends on the environment: with LC_ALL=C the same call fails with UnicodeEncodeError: '!!!' codec can't encode character '\xe9', because the codec is now ASCII.

name 3.15.2 3.16.0a2 (Linux, UTF-8 locale)
'!!!', '@', '()', 'é', '한글' LookupError codec for the locale encoding
'iconv:한글' LookupError codec for the locale encoding
',', '/', '//TRANSLIT', '//abc' LookupError codec for the locale encoding
'no-such-codec', '8', 'ü8' LookupError LookupError
Cause

iconv_search_function in Lib/encodings/__init__.py rejects only the empty name and then probes the name with _iconv_encode(name, ''). glibc's iconv_open() first removes trailing whitespace, / and , and any //-suffix such as //TRANSLIT (gconv_parse_code() in iconv/gconv_charset.c), then drops every remaining character that is not an ASCII alphanumeric, / or one of _-.,: (strip() in iconv/gconv_charset.h), and treats the resulting empty name as the charset of the current locale (__gconv_open in iconv/gconv_open.c: "If the string is empty define this to mean the charset of the currently selected locale"). So the probe succeeds for any name made only of dropped characters, and also for names whose charset part is empty once the suffix is removed, such as ',', '/', '//TRANSLIT' and '//abc'.

    def iconv_search_function(encoding):
        ...
        if not name:
            return None
        # Test if the encoding is supported by iconv.
        try:
            _iconv_encode(name, '')
        except (LookupError, OSError):
            return None
Possible fix

Before probing, reject a name unless the part before its first / or , contains an ASCII alphanumeric. Requiring an alphanumeric anywhere in the name is not enough: '//TRANSLIT' and '//abc' contain one and still resolve to the locale encoding.

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Linguagem predominante
Python
Estrelas
78.2k
Forks
36.6k
Merge médio
19h 57min
PRs com merge (30d)
630

Preparar o ambiente

Abrir no Codespaces

Inicia o contêiner de desenvolvimento do projeto no navegador, com a sua própria conta do GitHub.

Primeiros passos

  1. Leia a issue inteira e depois o guia de contribuição do projeto.
  2. Comente na issue dizendo que vai assumir — evita que duas pessoas façam o mesmo trabalho.
  3. Faça um fork do repositório e trabalhe em uma branch.
  4. Abra um pull request que referencie o número da issue.

Mais de python/cpython

Todas as issues de python/cpython

Issues semelhantes

Mais issues de Python

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.