lancedb/lancedb

Feature: Rust polars import / export

Aberta

#1.099 aberto em 13 de mar. de 2024

 (12 comentários) (1 reação) (0 responsável)HTML (876 forks)batch import
enhancementgood first issue

Métricas do repositório

Stars
 (10.303 estrelas)
Métricas de merge de PR
 (Mesclagem média 2d 17h) (82 fundiu PRs em 30d)

Description

SDK

Rust

Description

In the Rust SDK we should be able to go to/from polars more easily. For example, something like the following should work when adding data:

let my_df: DataFrame = ...
tbl.add(data_source::from_polars(my_df)).execute();

The above should also work with LazyFrame, although, for today at least, this will probably involve the consumption / execution of the LazyFrame:

let my_df: LazyFrame = ...
tbl.add(data_source::from_polars_lazy(my_df).execute();

Also note that DataSource doesn't exist yet. In my brainstorming its just a module with a bunch of static methods (enabled by feature flags) for creating Box<dyn RecordBatchReader + Send> from various inputs. As a result, this technique should also be usable in other methods that need data (e.g. create_table, merge_insert)

Guia do colaborador