Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

`codecs.StreamReader.read()` silently drops an incomplete character at the end of the stream

Aperta
  • Qualcosa non va?
#160,309 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

@saiyam-jain-dev ci sta già lavorando.

Dal 25/10/2026.

  • #160642 di @saiyam-jain-dev — aperta

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
56/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
python
Ambito
backend, testing

Direzione di ricerca

Start with Lib/codecs.py around the StreamReader.read() implementation and the linked expected-failure test StreamReaderTruncationBug.test_read_reports_truncated_input from PR #160155. Compare its end-of-stream behavior with IncrementalDecoder.decode(b'', final=True) and check the reported UTF-8, UTF-16, UTF-32, and escape-codec cases. Done means truncated input raises or uses the selected error handler instead of being silently discarded.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

stdlib type-bug

Bug description

StreamReader.read() decodes with the stateless codec function and keeps any undecoded trailing bytes for the next call. When the stream is exhausted there is no next call, so an incomplete final character is neither reported nor replaced; it just disappears:

>>> import codecs, io
>>> reader = codecs.getreader('utf-8')(io.BytesIO(b'ab\xe2'))
>>> reader.read()
'ab'
>>> reader.read()
''

Every other way of decoding the same bytes either raises or applies the error handler:

>>> b'ab\xe2'.decode('utf-8')
Traceback (most recent call last):
  File "<python-input-3>", line 1, in <module>
    b'ab\xe2'.decode('utf-8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe2 in position 2: unexpected end of data
>>> io.TextIOWrapper(io.BytesIO(b'ab\xe2'), encoding='utf-8').read()
Traceback (most recent call last):
  ...
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe2 in position 2: unexpected end of data
>>> codecs.getreader('utf-8')(io.BytesIO(b'ab\xe2'), 'replace').read()
'ab'
>>> b'ab\xe2'.decode('utf-8', 'replace')
'ab�'

So with errors='strict' a truncated file reads as if it were well-formed, and with 'replace' the replacement character that would mark the damage is missing. The same happens with utf-16, utf-32 and the escape codecs (unicode_escape with a trailing backslash); the CJK codecs are not affected, because their stateless decoders raise on an incomplete sequence themselves.

The cause is in read(): self.decode is the codec's stateless function with its default final=False (Lib/codecs.py line 512), and the loop exits on empty newdata without ever telling the decoder that the input is complete; self.bytebuffer keeps the fragment forever. Fixing it would mean a final decode pass once the underlying stream returns nothing, and raising (or applying the handler) if bytes remain, which is what IncrementalDecoder.decode(b'', final=True) does. The incremental decoders did not exist when StreamReader was written, which I assume is how this came about.

I realise codecs.open() is deprecated since 3.14 and StreamReader is on its way out with it, so feel free to say this is not worth touching; but it is silent data loss in the strict mode, which seemed worth having on record.

This is the same mechanism as #56717 (bpo-12508, 2011), which describes it for UTF-8 with errors='replace' and has had no activity since; I am filing it anyway because it applies to every codec with multi-byte characters and the strict-mode data loss is the part that matters, but feel free to fold this into that one.

Found by the Hypothesis tests in #160155, where it is pinned as StreamReaderTruncationBug.test_read_reports_truncated_input under expectedFailure.

CPython versions tested on

CPython main (6e1bd24e4d8)

Operating systems tested on

Linux

Lingua principale
Python
Stelle
77.7k
Fork
38.3k
Merge medio
12h 27m
PR unite (30g)
710

Preparare l'ambiente

Apri in Codespaces

Avvia il container di sviluppo del progetto nel browser, con il tuo account GitHub.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di python/cpython

Tutte le issue di python/cpython

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.