Hacktoberfest 2026：维护者为十月标记出来的 issue，仍然开放、适合新手。 浏览 Hacktoberfest issue

Use bounded streaming compression for artifact export instead of whole-snapshot buffers

未关闭
  • 哪里不对？
#2,577 1 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 2 天内回复

还没有人认领这个 Issue。

评估

难度
4/5
预计耗时
3-5 天
新手友好度
35/100
Issue 类型
重构
描述清晰度
基本清楚
活跃度
活跃
技术栈
c, sqlite
领域
performance

调研方向

Start in src/pipeline/artifact.c with prepare_snapshot_db, read_file_alloc and cbm_artifact_export, where the snapshot is read fully into memory and compressed with cbm_zstd_compress_bound and cbm_zstd_compress. Read the zstd wrapper to see how a streaming context would fit. Done means the export streams the snapshot in bounded chunks to a temp file, keeps atomic publication and cleanup, and the artifact imports with identical graph and metadata; check this with the export tests and an import round-trip.

由索引模型根据 Issue 内容生成。

描述

enhancement parsing/quality priority/high stability/performance ux/behavior

Request

Please consider bounded streaming compression for artifact export instead of buffering the entire uncompressed SQLite snapshot and a worst-case compressed output buffer simultaneously.

This is a public-source-based performance improvement request, not a claim of a newly executed runtime reproduction. All implementation references below are to this public repository.

Current implementation

In v0.11.0 src/pipeline/artifact.c:

  1. prepare_snapshot_db creates a SQLite snapshot and calls read_file_alloc(tmp_path, out_size).
  2. read_file_alloc allocates and reads the entire snapshot into memory.
  3. cbm_artifact_export calculates cbm_zstd_compress_bound(db_size), allocates the output buffer at that bound, and calls cbm_zstd_compress in one operation before freeing the input.

The same allocation pattern is present in main revision ec6f837296175a53cfd793ef4730189ab3d75268.

Why this matters

From the allocation structure, export buffer capacity scales with the entire uncompressed database size, even if the output artifact compresses very well. The input is read in full, while the output allocation reserves worst-case capacity. Allocated or reserved capacity should not be confused with physical resident memory; actual footprint depends on which pages are touched and allocator/OS behavior.

Changing the compression level alone would not remove the whole-file input allocation. Extraction-phase memory improvements also do not by themselves eliminate these export buffers.

Suggested implementation

  • Retain the existing consistent SQLite snapshot step.
  • Stream that snapshot from disk through a zstd streaming context into a temporary output file, using bounded input/output chunks.
  • Preserve atomic publication, metadata correctness, error handling, and cleanup of temporary artifacts.
  • Integrate export allocations into memory-budget accounting, or report an explicit failure if the configured budget cannot accommodate the required working set.

Suggested regression tests (not executed here)

Use isolated, synthetic, valid CBM database fixtures with increasing snapshot sizes, covering both highly repetitive and less compressible payloads:

  1. Call the real artifact export path and measure its memory separately from extraction/indexing.
  2. Verify the compression buffers stay bounded as snapshot size increases, distinguishing buffer allocation from end-to-end SQLite/OS footprint.
  3. Import the resulting artifacts and verify graph/metadata equivalence.
  4. Exercise write failures and cancellation where supported, ensuring failed exports do not replace the previous valid artifact or leave temporary files behind.

Related work / scope

  • #1953 / #2096 already address self-triggering watcher reindexing. This request concerns the resource cost of a single artifact export, not that fixed loop.
  • #2202 improves indexing memory behavior; this request targets the separate whole-snapshot compression path identified above.
  • #593 is the related large-repository performance/memory tracker.

I searched existing issues for artifact/memory and compression/memory and did not find a dedicated streaming-export request. Please link this to an existing item if there is one.

主要语言
C
星标
45.8k
派生
3.8k
平均合并
3 天 16 小时
30 天内合并 PR
203

环境准备

这个项目没有提供开发容器、Dockerfile 或贡献指南，环境需要你自己搭建：先看它的 README，通用步骤见我们的新手贡献指南。

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

DeusData/codebase-memory-mcp 的其他 Issue

查看 DeusData/codebase-memory-mcp 的全部 Issue

相似的 Issue

更多 C Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。