lance-format/lance

The ProjectionPlan::project_batch function is inefficient and noisy

Open

#5,069 建立於 2025年10月24日

在 GitHub 查看
 (2 留言) (0 反應) (1 負責人)Rust (695 fork)github user discovery
good first issue

倉庫指標

Star
 (6,582 star)
PR 合併指標
 (平均合併 6天 1小時) (30 天內合併 219 個 PR)

描述

In take workloads we utilize a function called ProjectionPlan::project_batch. This function creates a OneShotExec that reads in the batch and a ProjectExec that applies the projection expressions. This is probably fine if there are actually projection expressions to evaluate. However, if we all we are doing is reordering or dropping columns (or especially if the projection is an identity projections 🤦) we are adding quite a bit of unneeded overhead.

To put this into context, in a recent random access benchmarking effort I found this to be responsible for about 8% of the latency even though there was no actual projection (it was just identity).

貢獻者指南