lancedb/lancedb

Feature: to_lance method in asynchronous table python api

Open

#1.387 geöffnet am 19. Juni 2024

Auf GitHub ansehen
 (1 Kommentar) (1 Reaktion) (0 zugewiesene Personen)HTML (10.303 Stars) (876 Forks)batch import
Rustenhancementgood first issue

Beschreibung

SDK

Python

Description

Currently, in python api, "to_lance" method that returns a pyarrow dataset is only supported in synchronous table. I was wondering if "to_lance" method can be exposed for asynchronous tables as well.

For using third party tools like duckdb, pyarrow dataset is preferred over pyarrow tables as "Arrow Datasets allow for full column selection and filter pushdowns, unlike Arrow tables which are eagerly loaded into memory."

Since the newly introduced lance v2 is supported only in asynchronous tables , and since asynchronous tables do not have a to_lance method, we are not able to evaluate features like filter pushdowns, when using lance v2 tables against duckdb.

Contributor Guide

Tech Stack
python
Domain
api
Issue Type
feature
SchwierigkeitGeschätzte Implementierungsschwierigkeit für neue Contributors, von 1 für sehr kleine Änderungen bis 5 für Expertenarbeit.
2
Geschätzte ZeitEin grober Zeitrahmen, um zu recherchieren, zu implementieren, zu testen und einen Pull Request vorzubereiten.
1-3 hours
AktivitätsstatusWie verfügbar das Issue gerade wirkt: frisch, aktiv, stale, blockiert oder wartet auf Maintainer-Input.
fresh
KlarheitWie klar das Issue die erwartete Änderung, Akzeptanzkriterien und den nächsten Schritt erklärt.
clear
Voraussetzungen
Python async/awaitFamiliarity with LanceDB Python APIUnderstanding of existing to lance method
EinsteigerfreundlichkeitEin Score von 1 bis 100, der schätzt, wie zugänglich dieses Issue für First-Time Contributors ist.
80
Research-Richtung
Start by reviewing the existing `to lance` method in the synchronous table implementation (likely in `lancedb/table.py` or similar). Then examine the asynchronous table class (e.g., `AsyncTable`) to see its structure and how other methods are implemented. The issue mentions that Lance V2 is only available via async tables, so focus on adapting the sync logic to work with async contexts. Check if there are any comments or linked discussions for further context. Finally, implement the method ensuring it returns a Lance dataset as PyArrow dataset.