Use bounded streaming compression for artifact export instead of whole-snapshot buffers
メンテナーはふだん 2 日以内に返信
まだ誰も着手していません。
評価
- 難易度
- 4/5
- 見積もり時間
- 3〜5日
- 初心者へのやさしさ
- 35/100
- issue の種類
- リファクタリング
- 明瞭さ
- おおむね明確
- 活発さ
- 活発
- 技術スタック
- c, sqlite
- 領域
- performance
調査の方向性
Start in src/pipeline/artifact.c with prepare_snapshot_db, read_file_alloc and cbm_artifact_export, where the snapshot is read fully into memory and compressed with cbm_zstd_compress_bound and cbm_zstd_compress. Read the zstd wrapper to see how a streaming context would fit. Done means the export streams the snapshot in bounded chunks to a temp file, keeps atomic publication and cleanup, and the artifact imports with identical graph and metadata; check this with the export tests and an import round-trip.
索引モデルが issue の本文から書いたものです。
説明
Request
Please consider bounded streaming compression for artifact export instead of buffering the entire uncompressed SQLite snapshot and a worst-case compressed output buffer simultaneously.
This is a public-source-based performance improvement request, not a claim of a newly executed runtime reproduction. All implementation references below are to this public repository.
Current implementation
In v0.11.0 src/pipeline/artifact.c:
prepare_snapshot_dbcreates a SQLite snapshot and callsread_file_alloc(tmp_path, out_size).read_file_allocallocates and reads the entire snapshot into memory.cbm_artifact_exportcalculatescbm_zstd_compress_bound(db_size), allocates the output buffer at that bound, and callscbm_zstd_compressin one operation before freeing the input.
The same allocation pattern is present in main revision ec6f837296175a53cfd793ef4730189ab3d75268.
Why this matters
From the allocation structure, export buffer capacity scales with the entire uncompressed database size, even if the output artifact compresses very well. The input is read in full, while the output allocation reserves worst-case capacity. Allocated or reserved capacity should not be confused with physical resident memory; actual footprint depends on which pages are touched and allocator/OS behavior.
Changing the compression level alone would not remove the whole-file input allocation. Extraction-phase memory improvements also do not by themselves eliminate these export buffers.
Suggested implementation
- Retain the existing consistent SQLite snapshot step.
- Stream that snapshot from disk through a zstd streaming context into a temporary output file, using bounded input/output chunks.
- Preserve atomic publication, metadata correctness, error handling, and cleanup of temporary artifacts.
- Integrate export allocations into memory-budget accounting, or report an explicit failure if the configured budget cannot accommodate the required working set.
Suggested regression tests (not executed here)
Use isolated, synthetic, valid CBM database fixtures with increasing snapshot sizes, covering both highly repetitive and less compressible payloads:
- Call the real artifact export path and measure its memory separately from extraction/indexing.
- Verify the compression buffers stay bounded as snapshot size increases, distinguishing buffer allocation from end-to-end SQLite/OS footprint.
- Import the resulting artifacts and verify graph/metadata equivalence.
- Exercise write failures and cancellation where supported, ensuring failed exports do not replace the previous valid artifact or leave temporary files behind.
Related work / scope
- #1953 / #2096 already address self-triggering watcher reindexing. This request concerns the resource cost of a single artifact export, not that fixed loop.
- #2202 improves indexing memory behavior; this request targets the separate whole-snapshot compression path identified above.
- #593 is the related large-repository performance/memory tracker.
I searched existing issues for artifact/memory and compression/memory and did not find a dedicated streaming-export request. Please link this to an existing item if there is one.
- 主要言語
- C
- スター
- 45.8k
- フォーク
- 3.8k
- 平均マージ
- 3日 4時間
- マージ済み PR（30日）
- 208
環境構築
このプロジェクトには開発コンテナ、Dockerfile、コントリビューションガイドがありません。まず README を読み、一般的な手順ははじめてのコントリビューションガイドを参照してください。
はじめの一歩
- issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
- 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
- リポジトリをフォークし、ブランチを切って変更します。
- issue 番号を参照したプルリクエストを送ります。
DeusData/codebase-memory-mcp のほかの issue
-
bug editor/integration
難易度 2/5 1〜3時間 初心者へのやさしさ 72/100
DeusData/codebase-memory-mcp#2787 · コメント 1 件 ·
メンテナーはふだん 2 日以内に返信
-
editor/integration parsing/quality
難易度 2/5 1〜3時間 初心者へのやさしさ 68/100
DeusData/codebase-memory-mcp#2286 ·
メンテナーはふだん 2 日以内に返信
-
install writes an unsupported `args` key into the Claude Code SubagentStart hook, so it never runsオープンbug editor/integration priority/high stability/performance ux/behavior
難易度 2/5 1〜3時間 初心者へのやさしさ 72/100
DeusData/codebase-memory-mcp#2239 · コメント 1 件 ·
メンテナーはふだん 2 日以内に返信
-
bug priority/high ux/behavior
難易度 2/5 1〜3時間 初心者へのやさしさ 72/100
DeusData/codebase-memory-mcp#2232 · コメント 2 件 ·
メンテナーはふだん 2 日以内に返信
-
documentation editor/integration parsing/quality priority/normal
難易度 2/5 1〜3時間 初心者へのやさしさ 85/100
DeusData/codebase-memory-mcp#2169 ·
メンテナーはふだん 2 日以内に返信
DeusData/codebase-memory-mcp の issue をすべて見る
似ている issue
-
難易度 2/5 1〜3時間 初心者へのやさしさ 65/100
-
難易度 2/5 1〜3時間 初心者へのやさしさ 78/100
HexaCluster/credcheck#86 ·
-
難易度 2/5 1〜3時間 初心者へのやさしさ 66/100
mongodb/winkerberos#175 ·
-
bug
難易度 2/5 1〜3時間 初心者へのやさしさ 74/100
AOMediaCodec/SVT-AV1#3320 ·
-
difficulty: easy
難易度 2/5 1〜3時間 初心者へのやさしさ 64/100