[client-v2] Optimize RowBinaryFormatWriter by precompiling column serializers and reducing Object[] row-staging overhead

未关闭
  • 哪里不对？
#3,289 2 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 1 天内回复

还没有人认领这个 Issue。

评估

难度
4/5
预计耗时
3-5 天
新手友好度
22/100
Issue 类型
重构
描述清晰度
基本清楚
活跃度
活跃
技术栈
java
领域
performance

调研方向

Start with RowBinaryFormatWriter, especially the typed setters (setLong, setInteger, setDouble) that delegate to setValue(int, Object), and commitRow(), which walks the Object[] row and calls RowBinaryFormatSerializer.writeValuePreamble and SerializerUtils.serializeData. The issue proposes precomputing a per-column serialization plan at construction and avoiding boxing, so the design choice comes first. Done means a benchmark of a wide row workload shows the speedup without changing the serialized output.

由索引模型根据 Issue 内容生成。

描述

v2-feedback

Description
While testing RowBinaryFormatWriter for our use case, we identified several bottlenecks in its current serialization path. The writer initially took approximately 684 seconds to process the workload. After addressing these bottlenecks, the execution time decreased to approximately 462.47 seconds, representing a 32.39% reduction in runtime.

The main bottlenecks identified are:

Schema-dependent operations are repeated for every column of every row, even though the TableSchema does not change during the writer’s lifetime.
Primitive setters such as setInteger(), setLong(), setFloat(), and setDouble() delegate to setValue(int, Object), causing primitive boxing.
Values for the entire row are staged in an Object[] until commitRow() is called.
The stored values are later unboxed during serialization.
These operations add avoidable CPU, allocation, memory-access, and garbage-collection overhead for large, wide RowBinary workloads.

The proposed improvement is to compile and cache a reusable serialization plan when the writer is constructed. A primitive-specialized or sequential writing path should also be considered to avoid unnecessary Object[] staging, boxing, and unboxing.

Affected component
client-v2
com.clickhouse.client.api.data_formats.RowBinaryFormatWriter
com.clickhouse.client.api.data_formats.RowBinaryFormatSerializer
com.clickhouse.client.api.data_formats.internal.SerializerUtils

Current behavior
Typed setter methods eventually delegate to the generic object setter:

public void setLong(int colIndex, long value) {
setValue(colIndex, value);
}

public void setValue(int colIndex, Object value) {
row[colIndex - 1] = value;

if (!rowStarted) {
    rowStarted = true;
}

}
Because setValue() accepts Object, primitive values are autoboxed
The values are retained in an Object[] until commitRow() is called.
commitRow() then performs another traversal of the row:

List columnList = tableSchema.getColumns();

for (int i = 0; i < row.length; i++) {
ClickHouseColumn column = columnList.get(i);
if (column.hasDefault()&& column.getDefaultValue() != ClickHouseColumn.DefaultValue.DEFAULT) {
continue;
}

if (RowBinaryFormatSerializer.writeValuePreamble(out, defaultSupport, column, row[i])) {
    SerializerUtils.serializeData(out, row[i], column);
}

}

Potential bottlenecks
1: Primitive boxing
Typed methods such as setInteger(), setLong(), and setDouble() delegate to:
setValue(int colIndex, Object value)
This boxes primitives and stores them in an Object[], adding allocation, reference indirection, later unboxing, and possible GC overhead.

2: Multiple passes over every row

Each row is:

  • Written into Object[]
  • Traversed again by commitRow()
  • Cleared using Arrays.fill(row, null)
    A schema-ordered writer could serialize values directly without staging the complete row.

Repeated type dispatch

3: For every cell, commitRow() calls:

SerializerUtils.serializeData(out, row[i], column);
The column type is static, but the serializer path is selected repeatedly for every row.

4: Repeated schema metadata processing

commitRow() repeatedly performs:
tableSchema.getColumns();
columnList.get(i);
column.hasDefault();
column.getDefaultValue();

Writable columns and other static metadata could be resolved once when the writer is created.

主要语言
Java
星标
1.6k
派生
649
平均合并
1 天 20 小时
30 天内合并 PR
39

环境准备

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

ClickHouse/clickhouse-java 的其他 Issue

查看 ClickHouse/clickhouse-java 的全部 Issue

相似的 Issue

更多 Java Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。