lance-format/lance

Allow users to set a distance threshold to consider two vectors a "very similiar"

Open

#875 opened on May 17, 2023

View on GitHub
 (1 comment) (0 reactions) (0 assignees)Rust (695 forks)github user discovery
A-indexA-pythonenhancementgood first issuehelp wanted

Repository metrics

Stars
 (6,582 stars)
PR merge metrics
 (Avg merge 6d 1h) (219 merged PRs in 30d)

Description

Problem Statement

For example, I have lots of images that are resized jpgs and gifs -- the hashes are technically different, but the vector l2 distance is tiny then there's deduplicating things like images and watermarked images -- i also want this to be grouped together and to pick just one which basically uses the vector distance again, but with a looser threshold

Contributor guide

Research direction
Investigate the existing vector index code to find where distance thresholds are used and add a configurable parameter for a 'very similar' threshold.
Tech stack
rust
Domain
backenddatabaseai
Issue type
Feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-3 hours
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Clear
Prerequisites
RustGit
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
55