Reduce stage log Put allocation overhead in filestore persistence

Abierto
  • ¿Algo no está bien?
#7,026 1 comentario 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 2 días

@manenim ya está trabajando en esto.

Desde el 18/10/2026.

  • #7027 de @rootp1 — cerrado sin fusionar

Evaluación

Dificultad
3/5
Tiempo estimado
1-2 días
Aptitud para principiantes
58/100
Tipo de issue
Refactorización
Claridad
Bastante claro
Estado de actividad
Tranquilo
Stack tecnológico
go
Área
backend

Línea de trabajo

Comienza en pkg/app/server/stagelogstore/filestore.go leyendo stageLogFileStore.Put alrededor de las líneas 65-82 y su TODO de rendimiento. Añade el benchmark enfocado descrito en el issue para lf.Blocks crecientes, y después compara las asignaciones y el coste de escritura mientras compruebas el comportamiento de lectura existente. Se considera terminado cuando haya una mejora medible, se conserven el formato delimitado por líneas y el marcador EOL, y no se produzcan cambios de comportamiento no relacionados.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

What happened:

Summary

stageLogFileStore.Put rewrites the entire stage-log fragment on every save by JSON-encoding each LogBlock into a fresh in-memory buffer and then writing the full byte slice to the filestore. The code already carries a TODO for this hot path, and the current implementation scales allocations and latency with total block count on every write.

Affected Area

  • pkg/app/server/stagelogstore/filestore.go
  • stage log persistence in the control plane filestore path

What you expected to happen:

Large or frequently updated stage logs should be persisted with lower peak allocation overhead and an encoding path that does not materialize avoidable intermediate payloads for every block.

How to reproduce it:

  1. Open pkg/app/server/stagelogstore/filestore.go.
  2. Inspect stageLogFileStore.Put around the existing performance TODO.
  3. Note that each call loops over all lf.Blocks, marshals each block individually, appends it to a bytes.Buffer, and finally calls filestore.Put with the complete byte slice.
  4. Add a focused benchmark that grows lf.Blocks and repeatedly calls Put.
  5. Observe allocation count and write cost grow with total block count because the full fragment is rebuilt for every save.

Environment:

  • piped version: repository checkout on 2026-07-13
  • control-plane version: repository checkout on 2026-07-13
  • Others: validated by current code inspection; previous related issue #40 was closed as stale, not fixed

Technical Evidence

  • pkg/app/server/stagelogstore/filestore.go:65-82 rebuilds the payload in memory for every Put
  • The code includes TODO: Reduce the number of marshaling log blocks for improving performance
  • Upstream issue #40 tracked the same performance concern but was auto-closed as stale without an implementation change

Suspected Root Cause

The filestore path is implemented as full-object persistence, and Put currently uses per-block json.Marshal calls plus a whole-fragment buffer build before every write.

Proposed Direction

Start with a benchmark for realistic stage-log sizes, then refactor the write path to stream JSON encoding into the final buffer or otherwise reduce avoidable intermediate allocations while preserving the existing line-delimited on-disk format and EOL completion marker.

Acceptance Criteria

  • A benchmark captures baseline throughput and allocations for larger stage-log fragments
  • The write path measurably reduces allocations or latency compared with the current implementation
  • Existing read compatibility for stored stage logs is preserved
  • No unrelated stage-log behavior changes are introduced
Lenguaje dominante
Go
Estrellas
1.4k
Forks
391
Merge medio
4 d 1 h
PR fusionados (30 d)
58

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de pipe-cd/pipecd

Todos los issues de pipe-cd/pipecd

Issues similares

Más issues de Go

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.