Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

Use bounded streaming compression for artifact export instead of whole-snapshot buffers

Abierto
  • ¿Algo no está bien?
#2,577 1 comentario 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 2 días

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
35/100
Tipo de issue
Refactorización
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
c, sqlite
Área
performance

Línea de trabajo

Start in src/pipeline/artifact.c with prepare_snapshot_db, read_file_alloc and cbm_artifact_export, where the snapshot is read fully into memory and compressed with cbm_zstd_compress_bound and cbm_zstd_compress. Read the zstd wrapper to see how a streaming context would fit. Done means the export streams the snapshot in bounded chunks to a temp file, keeps atomic publication and cleanup, and the artifact imports with identical graph and metadata; check this with the export tests and an import round-trip.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

enhancement parsing/quality priority/high stability/performance ux/behavior

Request

Please consider bounded streaming compression for artifact export instead of buffering the entire uncompressed SQLite snapshot and a worst-case compressed output buffer simultaneously.

This is a public-source-based performance improvement request, not a claim of a newly executed runtime reproduction. All implementation references below are to this public repository.

Current implementation

In v0.11.0 src/pipeline/artifact.c:

  1. prepare_snapshot_db creates a SQLite snapshot and calls read_file_alloc(tmp_path, out_size).
  2. read_file_alloc allocates and reads the entire snapshot into memory.
  3. cbm_artifact_export calculates cbm_zstd_compress_bound(db_size), allocates the output buffer at that bound, and calls cbm_zstd_compress in one operation before freeing the input.

The same allocation pattern is present in main revision ec6f837296175a53cfd793ef4730189ab3d75268.

Why this matters

From the allocation structure, export buffer capacity scales with the entire uncompressed database size, even if the output artifact compresses very well. The input is read in full, while the output allocation reserves worst-case capacity. Allocated or reserved capacity should not be confused with physical resident memory; actual footprint depends on which pages are touched and allocator/OS behavior.

Changing the compression level alone would not remove the whole-file input allocation. Extraction-phase memory improvements also do not by themselves eliminate these export buffers.

Suggested implementation

  • Retain the existing consistent SQLite snapshot step.
  • Stream that snapshot from disk through a zstd streaming context into a temporary output file, using bounded input/output chunks.
  • Preserve atomic publication, metadata correctness, error handling, and cleanup of temporary artifacts.
  • Integrate export allocations into memory-budget accounting, or report an explicit failure if the configured budget cannot accommodate the required working set.

Suggested regression tests (not executed here)

Use isolated, synthetic, valid CBM database fixtures with increasing snapshot sizes, covering both highly repetitive and less compressible payloads:

  1. Call the real artifact export path and measure its memory separately from extraction/indexing.
  2. Verify the compression buffers stay bounded as snapshot size increases, distinguishing buffer allocation from end-to-end SQLite/OS footprint.
  3. Import the resulting artifacts and verify graph/metadata equivalence.
  4. Exercise write failures and cancellation where supported, ensuring failed exports do not replace the previous valid artifact or leave temporary files behind.

Related work / scope

  • #1953 / #2096 already address self-triggering watcher reindexing. This request concerns the resource cost of a single artifact export, not that fixed loop.
  • #2202 improves indexing memory behavior; this request targets the separate whole-snapshot compression path identified above.
  • #593 is the related large-repository performance/memory tracker.

I searched existing issues for artifact/memory and compression/memory and did not find a dedicated streaming-export request. Please link this to an existing item if there is one.

Lenguaje dominante
C
Estrellas
45.8k
Forks
3.8k
Merge medio
3 d 6 h
PR fusionados (30 d)
201

Preparar el entorno

Este proyecto no incluye contenedor de desarrollo, Dockerfile ni guía de contribución, así que la configuración corre por tu cuenta: empieza por su README y consulta nuestra guía para la primera contribución para los pasos generales.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de DeusData/codebase-memory-mcp

Todos los issues de DeusData/codebase-memory-mcp

Issues similares

Más issues de C

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.