[client-v2] Optimize RowBinaryFormatWriter by precompiling column serializers and reducing Object[] row-staging overhead

Aperta
  • Qualcosa non va?
#3,289 2 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
22/100
Tipo di issue
Refactoring
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
java
Ambito
performance

Direzione di ricerca

Start with RowBinaryFormatWriter, especially the typed setters (setLong, setInteger, setDouble) that delegate to setValue(int, Object), and commitRow(), which walks the Object[] row and calls RowBinaryFormatSerializer.writeValuePreamble and SerializerUtils.serializeData. The issue proposes precomputing a per-column serialization plan at construction and avoiding boxing, so the design choice comes first. Done means a benchmark of a wide row workload shows the speedup without changing the serialized output.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

v2-feedback

Description
While testing RowBinaryFormatWriter for our use case, we identified several bottlenecks in its current serialization path. The writer initially took approximately 684 seconds to process the workload. After addressing these bottlenecks, the execution time decreased to approximately 462.47 seconds, representing a 32.39% reduction in runtime.

The main bottlenecks identified are:

Schema-dependent operations are repeated for every column of every row, even though the TableSchema does not change during the writer’s lifetime.
Primitive setters such as setInteger(), setLong(), setFloat(), and setDouble() delegate to setValue(int, Object), causing primitive boxing.
Values for the entire row are staged in an Object[] until commitRow() is called.
The stored values are later unboxed during serialization.
These operations add avoidable CPU, allocation, memory-access, and garbage-collection overhead for large, wide RowBinary workloads.

The proposed improvement is to compile and cache a reusable serialization plan when the writer is constructed. A primitive-specialized or sequential writing path should also be considered to avoid unnecessary Object[] staging, boxing, and unboxing.

Affected component
client-v2
com.clickhouse.client.api.data_formats.RowBinaryFormatWriter
com.clickhouse.client.api.data_formats.RowBinaryFormatSerializer
com.clickhouse.client.api.data_formats.internal.SerializerUtils

Current behavior
Typed setter methods eventually delegate to the generic object setter:

public void setLong(int colIndex, long value) {
setValue(colIndex, value);
}

public void setValue(int colIndex, Object value) {
row[colIndex - 1] = value;

if (!rowStarted) {
    rowStarted = true;
}

}
Because setValue() accepts Object, primitive values are autoboxed
The values are retained in an Object[] until commitRow() is called.
commitRow() then performs another traversal of the row:

List columnList = tableSchema.getColumns();

for (int i = 0; i < row.length; i++) {
ClickHouseColumn column = columnList.get(i);
if (column.hasDefault()&& column.getDefaultValue() != ClickHouseColumn.DefaultValue.DEFAULT) {
continue;
}

if (RowBinaryFormatSerializer.writeValuePreamble(out, defaultSupport, column, row[i])) {
    SerializerUtils.serializeData(out, row[i], column);
}

}

Potential bottlenecks
1: Primitive boxing
Typed methods such as setInteger(), setLong(), and setDouble() delegate to:
setValue(int colIndex, Object value)
This boxes primitives and stores them in an Object[], adding allocation, reference indirection, later unboxing, and possible GC overhead.

2: Multiple passes over every row

Each row is:

  • Written into Object[]
  • Traversed again by commitRow()
  • Cleared using Arrays.fill(row, null)
    A schema-ordered writer could serialize values directly without staging the complete row.

Repeated type dispatch

3: For every cell, commitRow() calls:

SerializerUtils.serializeData(out, row[i], column);
The column type is static, but the serializer path is selected repeatedly for every row.

4: Repeated schema metadata processing

commitRow() repeatedly performs:
tableSchema.getColumns();
columnList.get(i);
column.hasDefault();
column.getDefaultValue();

Writable columns and other static metadata could be resolved once when the writer is created.

Lingua principale
Java
Stelle
1.6k
Fork
649
Merge medio
1g 20h
PR unite (30g)
39

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di ClickHouse/clickhouse-java

Tutte le issue di ClickHouse/clickhouse-java

Issue simili

Altre issue su Java

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.