perf(telemetry): remove the typed Parquet writer ingestion regression
维护者通常 1 天内回复
还没有人认领这个 Issue。
评估
- 难度
- 5/5
- 预计耗时
- 一周以上
- 新手友好度
- 35/100
- Issue 类型
- 缺陷
- 描述清晰度
- 基本清楚
- 活跃度
- 活跃
- 技术栈
- go
调研方向
Start with parquet_writer.go and the PR #272 review discussion, then run the transportbench workload with FANOUT_BENCH_PROFILE_DIR enabled. Compare alternating Linux ARM64 ingestion runs with CPU and post-load heap profiles, followed by mixed-p8 and cooldown RSS measurements. Done means the regression is addressed while preserving typed, nested, binary, residual, compaction, crash, corruption, ordering, and acknowledgement checks, with inspectable final artifacts.
由索引模型根据 Issue 内容生成。
描述
PR #272 review follow-up for parquet_writer.go: https://github.com/labstack/fanout/pull/272#discussion_r4157265075
The three alternating Linux ARM64 ingestion-only repetitions against #270 reproduced a roughly 15% durable-throughput regression. Optimize the measured typed writer/copy/allocation path while retaining format 3, typed values, nanosecond UTC columns, complete trace ordering, durable acknowledgements, and conservative admission bounds.
Median baseline -> DuckDB 2: 484,312 -> 411,796 acknowledged rows/s; 3,826 -> 4,123 Go allocated bytes/row; 1.316 -> 1.478 client+server CPU cores; 36.54 -> 42.09ms export p95. CPU cores rose about 12%; CPU time per acknowledged row rose about 32%, since throughput also declined. All six runs had zero export errors and passed storage verification. The fixture is 16 shared-transport clients, 1,000 rows/export, equal signals, 4 CPUs/6GiB, ten-second intervals, in-process clients/server, no dashboard queries. It is not the harness mixed-p8 workload. ValueBytes was about 2% cumulative CPU in the first current profile; its per-key charge is accounting, not allocated bytes.
Validation: repeat baseline/current alternating runs at least three times with CPU/post-load heap profiles, then repeat realistic mixed-p8 and cooldown RSS measurements. Preserve deep typed/nested/binary round trips, shredded residual types, compaction and crash/corruption checks. Profiles are generated with FANOUT_BENCH_PROFILE_DIR and the transportbench build tag; include inspectable reports/artifacts with the final measurements.
Implementation is authorized in PR #272; this issue tracks the fix and its validation.
- 主要语言
- Go
- 星标
- 11
- 派生
- 0
- 平均合并
- 7 小时 5 分钟
- 30 天内合并 PR
- 112
环境准备
- 提供 Dockerfile 或 Docker Compose 文件
- 有 Pull Request 模板
- 阅读贡献指南
从这里开始
- 先读完整个 Issue，再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库，在一个分支上完成修改。
- 提交 Pull Request，并在描述里引用这个 Issue 编号。
labstack/fanout 的其他 Issue
-
难度 2/5 1-3 小时 新手友好度 78/100
维护者通常 1 天内回复
-
bug
难度 2/5 1-3 小时 新手友好度 78/100
维护者通常 1 天内回复
-
难度 3/5 1-2 天 新手友好度 58/100
维护者通常 1 天内回复
-
难度 3/5 1-2 天 新手友好度 57/100
维护者通常 1 天内回复
-
难度 4/5 3-5 天 新手友好度 35/100
维护者通常 1 天内回复
相似的 Issue
-
难度 1/5 1 小时以内 新手友好度 72/100
open-feature/flagd-schemas#237 ·
-
enhancement
难度 2/5 1-3 小时 新手友好度 72/100
superplanehq/superplane#9234 ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 65/100
betterleaks/betterleaks#469 ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 82/100
prime-radiant-inc/evener#5901 ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 69/100