lance-format/lance

Merge with more than a single column for `left_on` possible?

Open

#1,211 opened on Sep 4, 2023

View on GitHub
 (2 comments) (0 reactions) (0 assignees)Rust (695 forks)github user discovery
enhancementgood first issuerust

Repository metrics

Stars
 (6,582 stars)
PR merge metrics
 (Avg merge 6d 1h) (219 merged PRs in 30d)

Description

@eddyxu @changhiskhan I had asked about adding columns in #767 and implementing this got put on hold, but now I have some questions.

Is the left_on and right_on only able to select a single column (it seems so)? We typically have datasets with inference_date and id, where for each day (inference_date there are multiple overlapping ids. We could make a column like date_id and join those two columns together to be able to do the merge. I am just checking if this feature will ever be possible or how best to design this from our end.

Thanks!

Contributor guide

Research direction
Investigate the current merge implementation in Lance, specifically the left on and right on parameters. Determine if the underlying storage supports multi column joins and propose a design for extending to multiple columns. Provide code examples or tests.
Tech stack
rust
Domain
backenddatabasedata engineering
Issue type
Feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-2 days
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Mostly clear
Prerequisites
RustDatabase concepts
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
30