lancedb/lancedb

Feature: Rust serde-based import / export

Open

#1,101 opened on Mar 13, 2024

View on GitHub
 (1 comment) (7 reactions) (0 assignees)HTML (876 forks)batch import
enhancementgood first issue

Repository metrics

Stars
 (10,303 stars)
PR merge metrics
 (Avg merge 2d 17h) (82 merged PRs in 30d)

Description

SDK

Rust

Description

Rust users are often unfamiliar with arrow and learning arrow can be a significant bottleneck to using lancedb. We could make it easier to import / export values by having the users create structs defining the record (similar to pydantic). In rust this is typically done with serde and there is a https://docs.rs/serde_arrow/latest/serde_arrow/ crate that we can investigate. The end result should be something like...

#[derive(Serialize, Deserialize, Debug)]
struct Location {
  x: f64,
  y: f64,
}

#[derive(Serialize, Deserialize, Debug)]
struct MyRecord {
  score: f32,
  caption: String,
  location: Location,
  clip: Vec<f64>,
}

...

tbl.add(data_source::from_structs(&[
  MyRecord { score: 32.0, caption: "my caption".to_string(), clip: vec![1.0, 2.0, 3.0], location: Location { x: 0.0, y: 0.0 } }
])).execute().await.unwrap();

Contributor guide