Hacktoberfest 2026: as issues que os mantenedores marcaram para outubro, abertas e boas para iniciantes. Ver issues do Hacktoberfest

The punycode incremental encoder and decoder transform each chunk separately

Aberta
  • Algo errado?
#160,308 0 comentários 0 reações 0 responsáveis Ver no GitHub

Mantenedores costumam responder em até 1 dia

Ninguém assumiu esta issue ainda.

  • #160599 de @OsmnvAslan — fechado sem integrar

Avaliação

Dificuldade
3/5
Tempo estimado
1-2 dias
Facilidade para iniciantes
72/100
Tipo de issue
Bug
Clareza
Claramente especificada
Status de atividade
Ativa
Stack de tecnologia
python
Domínio
backend, internationalization

Direção de pesquisa

Read Lib/encodings/punycode.py around lines 226–234 and the buffered incremental encoder and decoder base classes. Check how idna buffers input and how test_codecs.py marks punycode as broken in stateful mode. Implement buffering until final for both directions, then verify chunked input produces the same result as the whole-string transform.

Escrita pelo modelo de indexação a partir do texto da issue.

Descrição

stdlib type-bug

Bug description

encodings.punycode registers an IncrementalEncoder and IncrementalDecoder, but they just run the whole-string transform on every chunk they are given, so anything fed in more than one piece comes out wrong:

>>> import codecs
>>> 'ab'.encode('punycode')
b'ab-'
>>> encoder = codecs.getincrementalencoder('punycode')()
>>> encoder.encode('a') + encoder.encode('b', final=True)
b'a-b-'
>>> b'a-b-'.decode('punycode')
'a-b'

Punycode encodes the positions of the non-ASCII characters relative to the whole label, so a chunk cannot be encoded without the rest of the string; the same goes for decoding. The classes are two lines each, Lib/encodings/punycode.py lines 226-234, and ignore final altogether. This is half known: test_codecs.py lists punycode in broken_unicode_with_stateful ("encodings that don't work in stateful mode") and skips the chunked part of test_basics for it, but nothing says so in the docs, and the classes are still registered and handed out by codecs.getincrementalencoder(), codecs.iterencode() and StreamWriter.

I think the right fix is to buffer until final (the BufferedIncrementalEncoder and BufferedIncrementalDecoder base classes make that a few lines: _buffer_encode returns (b'', 0) until final), since that keeps codecs.iterencode, StreamWriter and friends working for anyone who hands them the whole label at once, which is presumably every current user. The idna codec already does something like it, buffering up to each ..

Related: #64331 (many incremental codecs do not handle fragmented data) mentions punycode in passing, in a list taken from that same test_codecs.py entry, but does not go into it.

Found by the Hypothesis tests in #160155, where it is pinned as PunycodeIncrementalBug.test_chunked_encode under expectedFailure; the chunked tests there leave punycode out otherwise.

CPython versions tested on

CPython main (6e1bd24e4d8)

Operating systems tested on

Linux

Linguagem predominante
Python
Estrelas
77.7k
Forks
38.3k
Merge médio
14h 29min
PRs com merge (30d)
699

Preparar o ambiente

Abrir no Codespaces

Inicia o contêiner de desenvolvimento do projeto no navegador, com a sua própria conta do GitHub.

Primeiros passos

  1. Leia a issue inteira e depois o guia de contribuição do projeto.
  2. Comente na issue dizendo que vai assumir — evita que duas pessoas façam o mesmo trabalho.
  3. Faça um fork do repositório e trabalhe em uma branch.
  4. Abra um pull request que referencie o número da issue.

Mais de python/cpython

Todas as issues de python/cpython

Issues semelhantes

Mais issues de Python

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.