The punycode incremental encoder and decoder transform each chunk separately
维护者通常 1 天内回复
还没有人认领这个 Issue。
- #160599 来自 @OsmnvAslan —— 已关闭，未合并
评估
- 难度
- 3/5
- 预计耗时
- 1-2 天
- 新手友好度
- 72/100
- Issue 类型
- 缺陷
- 描述清晰度
- 描述清楚
- 活跃度
- 活跃
- 技术栈
- python
调研方向
Read Lib/encodings/punycode.py around lines 226–234 and the buffered incremental encoder and decoder base classes. Check how idna buffers input and how test_codecs.py marks punycode as broken in stateful mode. Implement buffering until final for both directions, then verify chunked input produces the same result as the whole-string transform.
由索引模型根据 Issue 内容生成。
描述
Bug description
encodings.punycode registers an IncrementalEncoder and IncrementalDecoder, but they just run the whole-string transform on every chunk they are given, so anything fed in more than one piece comes out wrong:
>>> import codecs
>>> 'ab'.encode('punycode')
b'ab-'
>>> encoder = codecs.getincrementalencoder('punycode')()
>>> encoder.encode('a') + encoder.encode('b', final=True)
b'a-b-'
>>> b'a-b-'.decode('punycode')
'a-b'
Punycode encodes the positions of the non-ASCII characters relative to the whole label, so a chunk cannot be encoded without the rest of the string; the same goes for decoding. The classes are two lines each, Lib/encodings/punycode.py lines 226-234, and ignore final altogether. This is half known: test_codecs.py lists punycode in broken_unicode_with_stateful ("encodings that don't work in stateful mode") and skips the chunked part of test_basics for it, but nothing says so in the docs, and the classes are still registered and handed out by codecs.getincrementalencoder(), codecs.iterencode() and StreamWriter.
I think the right fix is to buffer until final (the BufferedIncrementalEncoder and BufferedIncrementalDecoder base classes make that a few lines: _buffer_encode returns (b'', 0) until final), since that keeps codecs.iterencode, StreamWriter and friends working for anyone who hands them the whole label at once, which is presumably every current user. The idna codec already does something like it, buffering up to each ..
Related: #64331 (many incremental codecs do not handle fragmented data) mentions punycode in passing, in a list taken from that same test_codecs.py entry, but does not go into it.
Found by the Hypothesis tests in #160155, where it is pinned as PunycodeIncrementalBug.test_chunked_encode under expectedFailure; the chunked tests there leave punycode out otherwise.
CPython versions tested on
CPython main (6e1bd24e4d8)
Operating systems tested on
Linux
- 主要语言
- Python
- 星标
- 77.7k
- 派生
- 38.3k
- 平均合并
- 13 小时 30 分钟
- 30 天内合并 PR
- 708
环境准备
在浏览器里用你自己的 GitHub 账号启动这个项目的开发容器。
- 没有 Dockerfile 或 Docker Compose 文件
- 有 Pull Request 模板
- 阅读贡献指南
从这里开始
- 先读完整个 Issue，再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库，在一个分支上完成修改。
- 提交 Pull Request，并在描述里引用这个 Issue 编号。
python/cpython 的其他 Issue
-
tarfile: extractfile(member).read() of a sparse member is slow due to repeated bytes concatenation未关闭performance stdlib
难度 2/5 1-3 小时 新手友好度 75/100
维护者通常 1 天内回复
-
tests topic-profiling type-bug
难度 2/5 1-3 小时 新手友好度 78/100
python/cpython#160334 · 1 条评论 ·
维护者通常 1 天内回复
-
`iso2022_jp_3` encodes U+9B1D but decodes the result as U+9B1C可能已有人在做 @kinqsradio 于 1 天前认领。 未关闭extension-modules type-bug
难度 2/5 1-3 小时 新手友好度 84/100
维护者通常 1 天内回复
-
interpreter-core type-feature
难度 1/5 1 小时以内 新手友好度 78/100
python/cpython#160109 · 1 条评论 ·
维护者通常 1 天内回复
-
extension-modules type-bug
难度 2/5 1-3 小时 新手友好度 72/100
python/cpython#160001 · 3 条评论 ·
维护者通常 1 天内回复
相似的 Issue
-
backlog
难度 1/5 1 小时以内 新手友好度 91/100
EchoTools/nevr-runtime#1108 ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 78/100
btclib-org/btclib-wallet#587 · 1 条评论 ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 75/100
-
难度 2/5 1-3 小时 新手友好度 72/100
维护者通常 1 天内回复
-
documentation
难度 2/5 1-3 小时 新手友好度 68/100
Judgment-Pack/judgment-pack-spec#275 ·
维护者通常 1 天内回复