Hacktoberfest 2026 : les issues que les mainteneurs ont marquées pour octobre, ouvertes et accessibles aux débutants. Parcourir les issues Hacktoberfest

`ZstdDecompressor` and `LZMADecompressor` raise on repeated `decompress(b'', max_length=0)` calls with output pending

Ouverte
  • Quelque chose ne va pas ?
#160,121 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub

Les mainteneurs répondent en général sous 1 jour

Personne n'a encore pris cette issue.

Évaluation

Difficulté
3/5
Temps estimé
1-2 jours
Accessibilité débutants
72/100
Type d'issue
Bug
Clarté
Clairement spécifiée
Activité
Active
Stack technique
python
Domaine
backend, performance

Piste de recherche

Start with Modules/_zstd/decompressor.c around the ZSTD_decompressStream call described in the issue, then inspect the LZMA decompressor path and the regression test test_max_length_zero_with_pending_output mentioned in PR #159960. Reproduce the repeated empty-input calls with max_length=0 on both backends. Done means both return b'' without an error while output is pending, with existing decompression behavior preserved.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Description

extension-modules type-bug

Bug description

When a compression.zstd.ZstdDecompressor has decompressed output waiting (because an earlier max_length cut it short), calling decompress with max_length=0 works 15 times and fails on the 16th:

>>> from compression import zstd
>>> d = zstd.ZstdDecompressor()
>>> d.decompress(zstd.compress(b"\x00"), max_length=0)
b''
>>> for _ in range(15):
...     d.decompress(b"", max_length=0)
...
>>> d.decompress(b"", max_length=0)
Traceback (most recent call last):
  File "<python-input-4>", line 1, in <module>
    d.decompress(b"", max_length=0)
_zstd.ZstdError: Unable to decompress Zstandard data: Operation made no progress over multiple calls, due to output buffer being full

Every one of those calls is a legitimate "buffer this, give me nothing yet" request, and needs_input is False throughout, so I would expect b'' each time with no error. The decompressor is not actually broken afterwards: a following decompress(b"", -1) still returns b"\x00" with eof set, so this is an error raised on a valid call rather than lost data.

LZMADecompressor fails the same sequence, on the fourth call rather than the sixteenth:

>>> import lzma
>>> d = lzma.LZMADecompressor()
>>> d.decompress(lzma.compress(b"\x00"), max_length=0)
b''
>>> for _ in range(3):
...     d.decompress(b"", max_length=0)
...
>>> d.decompress(b"", max_length=0)
Traceback (most recent call last):
  File "<python-input-4>", line 1, in <module>
    d.decompress(b"", max_length=0)
_lzma.LZMAError: Insufficient buffer space

That is the mechanism described in #160000 (liblzma's LZMA_BUF_ERROR leaking through when the output buffer is empty) showing up with input buffered rather than on a fresh decompressor; I expect the fix for that issue covers it, but it seemed worth recording the second shape since the fresh-decompressor test alone would not catch a regression here. BZ2Decompressor and zlib._ZlibDecompressor return b'' for any number of such calls.

The zstd mechanism: decompress_lock_held calls ZSTD_decompressStream even when max_length is 0, so with a zero-size output buffer: Modules/_zstd/decompressor.c lines 229-233. libzstd counts consecutive ZSTD_decompressStream calls that consume no input and produce no output (ZSTD_NO_FORWARD_PROGRESS_MAX, which is 16) and returns ZSTD_error_noForwardProgress_destFull once the limit is hit, as protection against callers looping forever. The simplest fix is probably to return b'' without calling into libzstd at all when max_length == 0, since there is nothing such a call can do except (with new input) append to the internal input buffer, which happens outside this loop anyway.

Found by running the Hypothesis tests from #159960 under Atheris, with the _zstd module compiled with -fsanitize=fuzzer-no-link. libFuzzer hit it within about twenty seconds, whereas plain Hypothesis had not found it in many runs, because it needs sixteen consecutive max_length=0 values; the zstd reproducer above is the shrunk result. Both backends are now pinned in that PR as test_max_length_zero_with_pending_output under expectedFailure.

CPython versions tested on

CPython main (39bc958bf9b), libzstd 1.5.7, liblzma 5.8.4

Operating systems tested on

Linux

cc @emmatyping

Langage dominant
Python
Étoiles
77.7k
Forks
38.3k
Merge moyen
14 h 29 min
PR mergées (30 j)
699

Préparer son environnement

Ouvrir dans Codespaces

Lance le conteneur de développement du projet dans votre navigateur, avec votre propre compte GitHub.

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Autres issues de python/cpython

Toutes les issues de python/cpython

Issues similaires

Plus d'issues Python

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.