[client-v2] Optimize RowBinaryFormatWriter by precompiling column serializers and reducing Object[] row-staging overhead

Aberta
  • Algo errado?
#3,289 2 comentários 0 reações 0 responsáveis Ver no GitHub

Mantenedores costumam responder em até 1 dia

Ninguém assumiu esta issue ainda.

Avaliação

Dificuldade
4/5
Tempo estimado
3-5 dias
Facilidade para iniciantes
22/100
Tipo de issue
Refatoração
Clareza
Razoavelmente clara
Status de atividade
Ativa
Stack de tecnologia
java
Domínio
performance

Direção de pesquisa

Start with RowBinaryFormatWriter, especially the typed setters (setLong, setInteger, setDouble) that delegate to setValue(int, Object), and commitRow(), which walks the Object[] row and calls RowBinaryFormatSerializer.writeValuePreamble and SerializerUtils.serializeData. The issue proposes precomputing a per-column serialization plan at construction and avoiding boxing, so the design choice comes first. Done means a benchmark of a wide row workload shows the speedup without changing the serialized output.

Escrita pelo modelo de indexação a partir do texto da issue.

Descrição

v2-feedback

Description
While testing RowBinaryFormatWriter for our use case, we identified several bottlenecks in its current serialization path. The writer initially took approximately 684 seconds to process the workload. After addressing these bottlenecks, the execution time decreased to approximately 462.47 seconds, representing a 32.39% reduction in runtime.

The main bottlenecks identified are:

Schema-dependent operations are repeated for every column of every row, even though the TableSchema does not change during the writer’s lifetime.
Primitive setters such as setInteger(), setLong(), setFloat(), and setDouble() delegate to setValue(int, Object), causing primitive boxing.
Values for the entire row are staged in an Object[] until commitRow() is called.
The stored values are later unboxed during serialization.
These operations add avoidable CPU, allocation, memory-access, and garbage-collection overhead for large, wide RowBinary workloads.

The proposed improvement is to compile and cache a reusable serialization plan when the writer is constructed. A primitive-specialized or sequential writing path should also be considered to avoid unnecessary Object[] staging, boxing, and unboxing.

Affected component
client-v2
com.clickhouse.client.api.data_formats.RowBinaryFormatWriter
com.clickhouse.client.api.data_formats.RowBinaryFormatSerializer
com.clickhouse.client.api.data_formats.internal.SerializerUtils

Current behavior
Typed setter methods eventually delegate to the generic object setter:

public void setLong(int colIndex, long value) {
setValue(colIndex, value);
}

public void setValue(int colIndex, Object value) {
row[colIndex - 1] = value;

if (!rowStarted) {
    rowStarted = true;
}

}
Because setValue() accepts Object, primitive values are autoboxed
The values are retained in an Object[] until commitRow() is called.
commitRow() then performs another traversal of the row:

List columnList = tableSchema.getColumns();

for (int i = 0; i < row.length; i++) {
ClickHouseColumn column = columnList.get(i);
if (column.hasDefault()&& column.getDefaultValue() != ClickHouseColumn.DefaultValue.DEFAULT) {
continue;
}

if (RowBinaryFormatSerializer.writeValuePreamble(out, defaultSupport, column, row[i])) {
    SerializerUtils.serializeData(out, row[i], column);
}

}

Potential bottlenecks
1: Primitive boxing
Typed methods such as setInteger(), setLong(), and setDouble() delegate to:
setValue(int colIndex, Object value)
This boxes primitives and stores them in an Object[], adding allocation, reference indirection, later unboxing, and possible GC overhead.

2: Multiple passes over every row

Each row is:

  • Written into Object[]
  • Traversed again by commitRow()
  • Cleared using Arrays.fill(row, null)
    A schema-ordered writer could serialize values directly without staging the complete row.

Repeated type dispatch

3: For every cell, commitRow() calls:

SerializerUtils.serializeData(out, row[i], column);
The column type is static, but the serializer path is selected repeatedly for every row.

4: Repeated schema metadata processing

commitRow() repeatedly performs:
tableSchema.getColumns();
columnList.get(i);
column.hasDefault();
column.getDefaultValue();

Writable columns and other static metadata could be resolved once when the writer is created.

Linguagem predominante
Java
Estrelas
1.6k
Forks
649
Merge médio
1d 20h
PRs com merge (30d)
39

Preparar o ambiente

Primeiros passos

  1. Leia a issue inteira e depois o guia de contribuição do projeto.
  2. Comente na issue dizendo que vai assumir — evita que duas pessoas façam o mesmo trabalho.
  3. Faça um fork do repositório e trabalhe em uma branch.
  4. Abra um pull request que referencie o número da issue.

Mais de ClickHouse/clickhouse-java

Todas as issues de ClickHouse/clickhouse-java

Issues semelhantes

Mais issues de Java

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.