perf(telemetry): remove the typed Parquet writer ingestion regression

Abierto
  • ¿Algo no está bien?
#277 2 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
35/100
Tipo de issue
Error
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
go
Área
data-engineering, observability, performance

Línea de trabajo

Start with parquet_writer.go and the PR #272 review discussion, then run the transportbench workload with FANOUT_BENCH_PROFILE_DIR enabled. Compare alternating Linux ARM64 ingestion runs with CPU and post-load heap profiles, followed by mixed-p8 and cooldown RSS measurements. Done means the regression is addressed while preserving typed, nested, binary, residual, compaction, crash, corruption, ordering, and acknowledgement checks, with inspectable final artifacts.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

PR #272 review follow-up for parquet_writer.go: https://github.com/labstack/fanout/pull/272#discussion_r4157265075

The three alternating Linux ARM64 ingestion-only repetitions against #270 reproduced a roughly 15% durable-throughput regression. Optimize the measured typed writer/copy/allocation path while retaining format 3, typed values, nanosecond UTC columns, complete trace ordering, durable acknowledgements, and conservative admission bounds.

Median baseline -> DuckDB 2: 484,312 -> 411,796 acknowledged rows/s; 3,826 -> 4,123 Go allocated bytes/row; 1.316 -> 1.478 client+server CPU cores; 36.54 -> 42.09ms export p95. CPU cores rose about 12%; CPU time per acknowledged row rose about 32%, since throughput also declined. All six runs had zero export errors and passed storage verification. The fixture is 16 shared-transport clients, 1,000 rows/export, equal signals, 4 CPUs/6GiB, ten-second intervals, in-process clients/server, no dashboard queries. It is not the harness mixed-p8 workload. ValueBytes was about 2% cumulative CPU in the first current profile; its per-key charge is accounting, not allocated bytes.

Validation: repeat baseline/current alternating runs at least three times with CPU/post-load heap profiles, then repeat realistic mixed-p8 and cooldown RSS measurements. Preserve deep typed/nested/binary round trips, shredded residual types, compaction and crash/corruption checks. Profiles are generated with FANOUT_BENCH_PROFILE_DIR and the transportbench build tag; include inspectable reports/artifacts with the final measurements.

Implementation is authorized in PR #272; this issue tracks the fix and its validation.

Lenguaje dominante
Go
Estrellas
11
Forks
0
Merge medio
7 h 5 min
PR fusionados (30 d)
112

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de labstack/fanout

Todos los issues de labstack/fanout

Issues similares

Más issues de Go

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.