Hacktoberfest 2026: as issues que os mantenedores marcaram para outubro, abertas e boas para iniciantes. Ver issues do Hacktoberfest

`ZstdDecompressor` and `LZMADecompressor` raise on repeated `decompress(b'', max_length=0)` calls with output pending

Aberta
  • Algo errado?
#160,121 0 comentários 0 reações 0 responsáveis Ver no GitHub

Mantenedores costumam responder em até 1 dia

Ninguém assumiu esta issue ainda.

Avaliação

Dificuldade
3/5
Tempo estimado
1-2 dias
Facilidade para iniciantes
72/100
Tipo de issue
Bug
Clareza
Claramente especificada
Status de atividade
Ativa
Stack de tecnologia
python
Domínio
backend, performance

Direção de pesquisa

Start with Modules/_zstd/decompressor.c around the ZSTD_decompressStream call described in the issue, then inspect the LZMA decompressor path and the regression test test_max_length_zero_with_pending_output mentioned in PR #159960. Reproduce the repeated empty-input calls with max_length=0 on both backends. Done means both return b'' without an error while output is pending, with existing decompression behavior preserved.

Escrita pelo modelo de indexação a partir do texto da issue.

Descrição

extension-modules type-bug

Bug description

When a compression.zstd.ZstdDecompressor has decompressed output waiting (because an earlier max_length cut it short), calling decompress with max_length=0 works 15 times and fails on the 16th:

>>> from compression import zstd
>>> d = zstd.ZstdDecompressor()
>>> d.decompress(zstd.compress(b"\x00"), max_length=0)
b''
>>> for _ in range(15):
...     d.decompress(b"", max_length=0)
...
>>> d.decompress(b"", max_length=0)
Traceback (most recent call last):
  File "<python-input-4>", line 1, in <module>
    d.decompress(b"", max_length=0)
_zstd.ZstdError: Unable to decompress Zstandard data: Operation made no progress over multiple calls, due to output buffer being full

Every one of those calls is a legitimate "buffer this, give me nothing yet" request, and needs_input is False throughout, so I would expect b'' each time with no error. The decompressor is not actually broken afterwards: a following decompress(b"", -1) still returns b"\x00" with eof set, so this is an error raised on a valid call rather than lost data.

LZMADecompressor fails the same sequence, on the fourth call rather than the sixteenth:

>>> import lzma
>>> d = lzma.LZMADecompressor()
>>> d.decompress(lzma.compress(b"\x00"), max_length=0)
b''
>>> for _ in range(3):
...     d.decompress(b"", max_length=0)
...
>>> d.decompress(b"", max_length=0)
Traceback (most recent call last):
  File "<python-input-4>", line 1, in <module>
    d.decompress(b"", max_length=0)
_lzma.LZMAError: Insufficient buffer space

That is the mechanism described in #160000 (liblzma's LZMA_BUF_ERROR leaking through when the output buffer is empty) showing up with input buffered rather than on a fresh decompressor; I expect the fix for that issue covers it, but it seemed worth recording the second shape since the fresh-decompressor test alone would not catch a regression here. BZ2Decompressor and zlib._ZlibDecompressor return b'' for any number of such calls.

The zstd mechanism: decompress_lock_held calls ZSTD_decompressStream even when max_length is 0, so with a zero-size output buffer: Modules/_zstd/decompressor.c lines 229-233. libzstd counts consecutive ZSTD_decompressStream calls that consume no input and produce no output (ZSTD_NO_FORWARD_PROGRESS_MAX, which is 16) and returns ZSTD_error_noForwardProgress_destFull once the limit is hit, as protection against callers looping forever. The simplest fix is probably to return b'' without calling into libzstd at all when max_length == 0, since there is nothing such a call can do except (with new input) append to the internal input buffer, which happens outside this loop anyway.

Found by running the Hypothesis tests from #159960 under Atheris, with the _zstd module compiled with -fsanitize=fuzzer-no-link. libFuzzer hit it within about twenty seconds, whereas plain Hypothesis had not found it in many runs, because it needs sixteen consecutive max_length=0 values; the zstd reproducer above is the shrunk result. Both backends are now pinned in that PR as test_max_length_zero_with_pending_output under expectedFailure.

CPython versions tested on

CPython main (39bc958bf9b), libzstd 1.5.7, liblzma 5.8.4

Operating systems tested on

Linux

cc @emmatyping

Linguagem predominante
Python
Estrelas
77.7k
Forks
38.3k
Merge médio
11h 58min
PRs com merge (30d)
672

Preparar o ambiente

Abrir no Codespaces

Inicia o contêiner de desenvolvimento do projeto no navegador, com a sua própria conta do GitHub.

Primeiros passos

  1. Leia a issue inteira e depois o guia de contribuição do projeto.
  2. Comente na issue dizendo que vai assumir — evita que duas pessoas façam o mesmo trabalho.
  3. Faça um fork do repositório e trabalhe em uma branch.
  4. Abra um pull request que referencie o número da issue.

Mais de python/cpython

Todas as issues de python/cpython

Issues semelhantes

Mais issues de Python

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.