apache/arrow-java
Performance concern: fillHoles() method and read buffer expansion efficiency.
オープン
#599 opened on 2025/02/10
Type: enhancementhelp wanted
Repository metrics
- Stars
- (89 個のスター)
- PR merge metrics
- (PR metrics pending)
説明
Hi everyone,
I have the following use case: I’m benchmarking the read throughput performance when dealing with a large number of non-dictionary string columns (300 columns). Based on the profiler output (see the attached picture), I’ve noticed that a significant amount of time is spent in the fillHoles() method, which is part of the read buffer expansion process.
My question is: why is the buffer filled one element at a time instead of using a bulk operation? Wouldn’t a batch approach be more efficient?
Looking forward to your insights. Thanks!