lance-format/lance
在 GitHub 查看The ProjectionPlan::project_batch function is inefficient and noisy
Open
#5,069 建立於 2025年10月24日
good first issue
倉庫指標
- Star
- (6,582 star)
- PR 合併指標
- (平均合併 6天 1小時) (30 天內合併 219 個 PR)
描述
In take workloads we utilize a function called ProjectionPlan::project_batch. This function creates a OneShotExec that reads in the batch and a ProjectExec that applies the projection expressions. This is probably fine if there are actually projection expressions to evaluate. However, if we all we are doing is reordering or dropping columns (or especially if the projection is an identity projections 🤦) we are adding quite a bit of unneeded overhead.
To put this into context, in a recent random access benchmarking effort I found this to be responsible for about 8% of the latency even though there was no actual projection (it was just identity).