lance-format/lance

The ProjectionPlan::project_batch function is inefficient and noisy

Open

#5,069 opened on 2025年10月24日

GitHub で見る
 (2 comments) (0 reactions) (1 assignee)Rust (695 forks)github user discovery
good first issue

Repository metrics

Stars
 (6,582 stars)
PR merge metrics
 (平均マージ 6d 1h) (30d で 219 merged PRs)

説明

In take workloads we utilize a function called ProjectionPlan::project_batch. This function creates a OneShotExec that reads in the batch and a ProjectExec that applies the projection expressions. This is probably fine if there are actually projection expressions to evaluate. However, if we all we are doing is reordering or dropping columns (or especially if the projection is an identity projections 🤦) we are adding quite a bit of unneeded overhead.

To put this into context, in a recent random access benchmarking effort I found this to be responsible for about 8% of the latency even though there was no actual projection (it was just identity).

コントリビューターガイド