perf(telemetry): remove the typed Parquet writer ingestion regression

Aperta
  • Qualcosa non va?
#277 2 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
35/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
go
Ambito
data-engineering, observability, performance

Direzione di ricerca

Start with parquet_writer.go and the PR #272 review discussion, then run the transportbench workload with FANOUT_BENCH_PROFILE_DIR enabled. Compare alternating Linux ARM64 ingestion runs with CPU and post-load heap profiles, followed by mixed-p8 and cooldown RSS measurements. Done means the regression is addressed while preserving typed, nested, binary, residual, compaction, crash, corruption, ordering, and acknowledgement checks, with inspectable final artifacts.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

PR #272 review follow-up for parquet_writer.go: https://github.com/labstack/fanout/pull/272#discussion_r4157265075

The three alternating Linux ARM64 ingestion-only repetitions against #270 reproduced a roughly 15% durable-throughput regression. Optimize the measured typed writer/copy/allocation path while retaining format 3, typed values, nanosecond UTC columns, complete trace ordering, durable acknowledgements, and conservative admission bounds.

Median baseline -> DuckDB 2: 484,312 -> 411,796 acknowledged rows/s; 3,826 -> 4,123 Go allocated bytes/row; 1.316 -> 1.478 client+server CPU cores; 36.54 -> 42.09ms export p95. CPU cores rose about 12%; CPU time per acknowledged row rose about 32%, since throughput also declined. All six runs had zero export errors and passed storage verification. The fixture is 16 shared-transport clients, 1,000 rows/export, equal signals, 4 CPUs/6GiB, ten-second intervals, in-process clients/server, no dashboard queries. It is not the harness mixed-p8 workload. ValueBytes was about 2% cumulative CPU in the first current profile; its per-key charge is accounting, not allocated bytes.

Validation: repeat baseline/current alternating runs at least three times with CPU/post-load heap profiles, then repeat realistic mixed-p8 and cooldown RSS measurements. Preserve deep typed/nested/binary round trips, shredded residual types, compaction and crash/corruption checks. Profiles are generated with FANOUT_BENCH_PROFILE_DIR and the transportbench build tag; include inspectable reports/artifacts with the final measurements.

Implementation is authorized in PR #272; this issue tracks the fix and its validation.

Lingua principale
Go
Stelle
11
Fork
0
Merge medio
7h 5m
PR unite (30g)
112

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di labstack/fanout

Tutte le issue di labstack/fanout

Issue simili

Altre issue su Go

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.