Hacktoberfest 2026：维护者为十月标记出来的 issue，仍然开放、适合新手。 浏览 Hacktoberfest issue

Provide an efficient way to decompress a sequence of chunks compressed with ZstdCompressionChunker

未关闭
  • 哪里不对？
#259 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 10 天内回复

还没有人认领这个 Issue。

评估

难度
5/5
预计耗时
一周以上
新手友好度
35/100
Issue 类型
功能
描述清晰度
基本清楚
活跃度
停滞
技术栈
c, python
领域
api, performance

调研方向

首先检查 ZstdCompressionChunker、ZstdDecompressor.read_to_iter，以及 issue 中精简后的 memoryview 复现代码。确定现有的受支持 API 是否能够增量处理压缩 chunk；否则，定义一种高效的逐 chunk 处理方案，避免拼接输入或输出，并且不会发生段错误，然后为报告的行为添加覆盖测试。

由索引模型根据 Issue 内容生成。

描述

My program wants to compress some large cached strings and decompress them later. I have no particular requirements on the form of the compressed data, so I used ZstdCompressionChunker to do the compression to avoid repeated reallocation of the output buffer. I would like to process the decompressed data in chunks to reduce peak memory usage. However there is no obvious efficient way to decompress chunks to chunks:

  • The ZstdCompressionChunker round-trip tests all concatenate the chunks with bytes.join for one-shot decompression. (Fine, they're tests.)

  • I tried chain.from_iterable(dctx.read_to_iter(c) for c in chunks). This doesn't work because each read_to_iter iterator expects to process a full stream. (I expected it to hold state in the ZstdDecompressor it was obtained from.)

  • ZstdCompressionObj's documentation says it isn't efficient:

    Because calls to decompress() may need to perform multiple memory (re)allocations, this streaming decompression API isn’t as efficient as other APIs.

  • read_to_iter's documentation says

    read_to_iter() accepts an object with a read(size) method that will return compressed bytes or an object conforming to the buffer protocol.

    so I wrote a class with a read method that returns memoryviews over the chunks (to avoid copying slices). The documentation is grammatically ambiguous; it turns out that read_to_iter segfaults (!) when given an object with a read method that returns an object conforming to the buffer protocol that is not exactly bytes (reduced test case below).

My feature request is to provide an efficient way to decompress a sequence of chunks compressed with ZstdCompressionChunker (or to document an existing method as the efficient way, if there is one).

import zstandard as zstd
b = b'AB' * 1000
d = zstd.compress(b)
assert zstd.decompress(memoryview(d)) == b # passes
class Whatever:
    def __init__(self, data):
        self.data = data
    def read(self, size):
        assert len(data) <= size
        return memoryview(self.data)
dctx = zstd.ZstdDecompressor()
assert b''.join(dctx.read_to_iter(Whatever(d))) == b # segfault

Segfaults using Arch Linux's python 3.13.2-1 and python-zstandard 0.23.0-2.

主要语言
C
星标
642
派生
118
平均合并
9 天 3 小时
30 天内合并 PR
6

环境准备

  • 没有 Dockerfile 或 Docker Compose 文件
  • 没有 Pull Request 模板
  • 阅读贡献指南

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

indygreg/python-zstandard 的其他 Issue

查看 indygreg/python-zstandard 的全部 Issue

相似的 Issue

更多 C Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。