ISO-2022 encoders accept U+001B (and `iso2022_kr` U+000E/U+000F) although the encoding cannot represent them
維護者通常 1 天內回覆
還沒有人認領這個 Issue。
評估
研究方向
Start with the ASCII fast path in Modules/cjkcodecs/_codecs_iso2022.c around lines 190–200, then review the Hypothesis regression in pull request #160155, where RoundTripBugs.test_iso2022_escape is marked as an expected failure. Check the affected ISO-2022 encoders, including the SO/SI cases for iso2022_kr, and confirm the regression and error-handler tests pass when these unrepresentable control characters are rejected.
由索引模型根據 Issue 內容生成。
描述
Bug description
In ISO 2022, the bytes 0x1B (ESC), 0x0E (SO) and 0x0F (SI) are special control characters that aren't supposed to exist by themselves: ESC introduces the designation sequences that switch character sets (ESC $ B selects JIS X 0208, and so on), and iso2022_kr uses SO and SI to shift between ASCII and KS X 1001. There is no way to put the characters U+001B, U+000E or U+000F into an ISO-2022-JP or ISO-2022-KR byte stream; a 0x1B byte in the output is the start of an escape sequence to whoever reads it.
I would expect that attempting to encode these by themselves (or in some other invalid configuration) to be a failure state and raise UnicodeEncodeError (allowing error handlers do their thing), which is what they do for other characters the encoding has no bytes for.
What they actually do is write the control character through as a bare byte, because the encoder's ASCII fast path accepts every code point below 0x80, Modules/cjkcodecs/_codecs_iso2022.c lines 190-200. The output is not valid ISO 2022, and the matching decoder either rejects it or reads it as something else. Alone, the byte is an incomplete escape sequence:
>>> '\x1b'.encode('iso2022_jp')
b'\x1b'
>>> b'\x1b'.decode('iso2022_jp')
Traceback (most recent call last):
File "<python-input-2>", line 1, in <module>
b'\x1b'.decode('iso2022_jp')
~~~~~~~~^^^^^^^^^^^^^^
UnicodeDecodeError: 'iso2022_jp' codec can't decode byte 0x1b in position 0: incomplete multibyte sequence
Followed by the right ASCII characters, it is an escape sequence, and the text that follows is decoded in the wrong character set:
>>> 'x\x1b$B$"'.encode('iso2022_jp')
b'x\x1b$B$"'
>>> _.decode('iso2022_jp')
'xあ'
And in front of a real designation sequence it makes the decoder consume that sequence as text:
>>> '\x1b\xa0'.encode('iso2022_jp_2004')
b'\x1b\x1b$(Q)"\x1b(B'
>>> _.decode('iso2022_jp_2004')
'\x1b\x1b$(Q)"'
Every iso2022_* codec does this for U+001B. iso2022_kr does the same with SO and SI, which its decoder then interprets as shifts:
>>> '\x0e'.encode('iso2022_kr')
b'\x0e'
>>> _.decode('iso2022_kr')
''
>>> 'a\x0f가'.encode('iso2022_kr').decode('iso2022_kr')
'a가'
I do not think this can be fixed on the decoding side: the second example shows that a decoder cannot tell a stray ESC from a real one, so only refusing to encode these characters makes the round trip hold. Raising is a behavior change for text that currently encodes without complaint, but what it currently encodes to is misread by any ISO-2022 reader, not just Python's. For comparison, hz has the same problem with ~ and escapes it as ~~; the other CJK codecs pass ESC through, but for them it is an ordinary control character on both sides, so they round-trip.
Found by the Hypothesis tests in #160155, where it is pinned as RoundTripBugs.test_iso2022_escape under expectedFailure; the round-trip and error-handler tests there keep these characters out of the generated text for these codecs until it is fixed.
- 主要語言
- Python
- 星號
- 77.7k
- 分支
- 38.3k
- 平均合併
- 11 小時 58 分鐘
- 30 天內合併 PR
- 672
環境準備
在瀏覽器裡用你自己的 GitHub 帳號啟動這個專案的開發容器。
- 沒有 Dockerfile 或 Docker Compose 檔案
- 有 Pull Request 範本
- 閱讀貢獻指南
從這裡開始
- 先讀完整個 Issue，再讀專案的貢獻指南。
- 在 Issue 下留言說明你要接手 —— 這能避免兩個人做同樣的事。
- Fork 儲存庫，在一個分支上完成修改。
- 送出 Pull Request，並在描述裡引用這個 Issue 編號。
python/cpython 的其他 Issue
-
topic-socket type-bug
難度 2/5 1-3 小時 新手友好度 72/100
python/cpython#160561 · 7 則留言 ·
維護者通常 1 天內回覆
-
tests topic-profiling type-bug
難度 2/5 1-3 小時 新手友好度 78/100
python/cpython#160334 · 1 則留言 ·
維護者通常 1 天內回覆
-
extension-modules type-bug
難度 2/5 1-3 小時 新手友好度 84/100
維護者通常 1 天內回覆
-
interpreter-core type-feature
難度 1/5 1 小時以內 新手友好度 78/100
python/cpython#160109 · 1 則留言 ·
維護者通常 1 天內回覆
-
extension-modules type-bug
難度 2/5 1-3 小時 新手友好度 72/100
python/cpython#160001 · 3 則留言 ·
維護者通常 1 天內回覆
相似的 Issue
-
bug P2 rust
難度 2/5 1-3 小時 新手友好度 82/100
維護者通常 1 天內回覆
-
enhancement
難度 2/5 1-3 小時 新手友好度 72/100
jscott3201/rusty-bacnet#2085 ·
維護者通常 1 天內回覆
-
難度 2/5 1-3 小時 新手友好度 84/100
evarness-ai/iris-harness#401 ·
-
難度 2/5 1-3 小時 新手友好度 76/100
sistemicorp/scripts#62 ·
-
area/mac help wanted needs-qa
難度 1/5 1 小時以內 新手友好度 76/100