Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

`iso2022_jp_3` encodes U+9B1D but decodes the result as U+9B1C

Open Beginner friendly
  • Is something wrong?
#160,306 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
84/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
python
Domain
internationalization

Research direction

Read Modules/cjkcodecs/_codecs_iso2022.c around jisx0213_2000_2_decoder and compare its plane 2 mapping logic with the plane 1 decoder. The issue identifies the missing else and the affected cell; use the RoundTripBugs.test_iso2022_jp_3_mapping test from PR #160155 as the regression check. Done means iso2022_jp_3 round trips U+9B1D correctly while retaining the expected 2000 mapping behavior.

Written by the indexing model from the issue text.

Description

extension-modules type-bug

Bug description

>>> '鬝'.encode('iso2022_jp_3')
b'\x1b$(P};\x1b(B'
>>> _.decode('iso2022_jp_3')
'鬜'
>>> hex(ord(_))
'0x9b1c'

JIS X 0213:2004 changed plane 2 row 93 cell 27 (0x7D3B) from U+9B1D to U+9B1C, and CPython emulates the 2000 mapping in the codecs named after that edition. euc_jisx0213 and shift_jisx0213 do so in both directions:

>>> '鬝'.encode('euc_jisx0213').decode('euc_jisx0213')
'鬝'
>>> '鬜'.encode('euc_jisx0213')
Traceback (most recent call last):
  File "<python-input-5>", line 1, in <module>
    '鬜'.encode('euc_jisx0213')
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
UnicodeEncodeError: 'euc_jisx0213' codec can't encode character '鬜' in position 0: illegal multibyte sequence

iso2022_jp_3 only does it when encoding: it accepts U+9B1D and rejects U+9B1C like the other 2000 codecs, but decodes 0x7D3B as U+9B1C, as iso2022_jp_2004 does.

The cause is in jisx0213_2000_2_decoder, Modules/cjkcodecs/_codecs_iso2022.c lines 728-739. The EMULATE_JISX0213_2000_DECODE_PLANE2_CHAR macro expands to an if that sets u = 0x9B1D for this cell, but the table lookup that follows it is a plain if (TRYMAP_DEC_ST(jisx0213_2_bmp, ...)) rather than else if, so it runs regardless and overwrites u with the 2004 value. The plane 1 decoder just above uses else if after its emulation macro and is fine, as are the EUC and Shift_JIS codecs, which use the OUTCHAR form of the macro in _codecs_jp.c. So the fix is the missing else.

A scan of every code point through iso2022_jp_3 found no other character that comes back different, so it should be just this one cell.

Found by the Hypothesis tests in #160155, where it is pinned as RoundTripBugs.test_iso2022_jp_3_mapping under expectedFailure and kept out of the generated text otherwise.

CPython versions tested on

CPython main (6e1bd24e4d8)

Operating systems tested on

Linux

Dominant language
Python
Stars
77.7k
Forks
38.3k
Avg merge
12h 27m
Merged PRs (30d)
645

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from python/cpython

All issues in python/cpython

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.