io: TextIOWrapper silently corrupts output when encoder is a no-op
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 75/100
Research direction
Read _io_TextIOWrapper_write_impl in Modules/_io/textio.c, especially the check linked in the report, and inspect nearby TextIOWrapper tests for encoder return types. Reproduce the passthrough encoder case from the issue, then add a regression test and verify that non-ASCII input raises the stated TypeError while the ASCII fast path still works.
Written by the indexing model from the issue text.
Description
Bug report
Summary
When an encoder returns the input str itself, TextIOWrapper.write treats it as ASCII and writes the str's internal storage to the file. For non-ASCII text the file is silently corrupted.
Expected: TypeError: encoder should return a bytes object, not 'str'.
Reproducer
We use a 44-character string that ends in an emoji, so CPython internally stores it as four bytes per character.
import codecs
class Enc(codecs.IncrementalEncoder):
def encode(self, text, final=False):
return text
info = codecs.CodecInfo(None, None, incrementalencoder=Enc,
incrementaldecoder=codecs.getincrementaldecoder("utf-8"),
name="passthrough")
codecs.register(lambda n: info if n == "passthrough" else None)
with open("out.txt", "w", encoding="passthrough") as f:
f.write("The quick brown fox jumps over the lazy dog🐍")
print(open("out.txt").read())
The quick b
Only the first 44 bytes (out of 44*4 = 176) were written to the file.
Root cause
Modules/_io/textio.c, in _io_TextIOWrapper_write_impl:
The exemption for b == text is meant for the ASCII fast path, which sets b = text only for ASCII text.
A Python encoder that returns its input also satisfies it, with any text.
Suggested fix
Tie the exemption to what the fast path relies on, that the text is ASCII:
if (!PyBytes_Check(b) && !(b == text && PyUnicode_IS_ASCII(text))) {
With this the reproducer raises the TypeError above.
CPython versions tested on:
CPython main branch
Operating systems tested on:
Linux
- Dominant language
- Python
- Stars
- 77.9k
- Forks
- 37.1k
- Avg merge
- 17h 43m
- Merged PRs (30d)
- 675
Getting set up
Starts the project's dev container in your browser, under your own GitHub account.
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from python/cpython
-
OS-windows performance stdlib type-feature
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
Maintainers usually reply within 1 day
-
OS-windows performance stdlib type-feature
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
Maintainers usually reply within 1 day
-
build OS-unsupported OS-windows type-bug
Difficulty 2/5 1-3 hours Newbie friendliness 60/100
Maintainers usually reply within 1 day
-
What's New in 3.12 credits the unstable C API tier to PEP 697 instead of PEP 689Possibly taken @TheLouisMahdi claimed this 4 days ago. Opendocs
Difficulty 1/5 Under an hour Newbie friendliness 82/100
python/cpython#161078 · 3 comments ·
Maintainers usually reply within 1 day
-
OS-windows type-bug
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
Maintainers usually reply within 1 day
Similar issues
-
docs size:S
Difficulty 1/5 Under an hour Newbie friendliness 88/100
Maintainers usually reply within 1 day
-
documentation
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
strawberry-graphql/strawberry#4794 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
dataforgoodfr/cetolog#21 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
btclib-org/btclib#2932 ·
Maintainers usually reply within 1 day