lance-format/lance

Support Arrow Tensor type

Open

#108 建立於 2022年8月15日

在 GitHub 查看
 (3 留言) (0 反應) (1 負責人)Rust (695 fork)github user discovery
A-pythonarrowenhancementhelp wantedrust

倉庫指標

Star
 (6,582 star)
PR 合併指標
 (平均合併 6天 1小時) (30 天內合併 219 個 PR)

描述

edit: use arrow::extension::FixedShapeTensorType

https://arrow.apache.org/docs/dev/format/CanonicalExtensions.html#fixed-shape-tensor-extension

Problem

In arrow, Tensors do not have corresponding Array type, as it can not be used to store in a Table / Dataset / RecordBatch.

To support ML datasets, it is desirable to have tensors stored within the datasets.

Desired Behavior

To support parametrized Tensors in a Table / Datasets. A TensorArray and TensorType should allow parameters, i.e., shade, data type (i.e., bid_width) and etc.

class TensorType {
   TensorType(data_type, shade, ...)
}

貢獻者指南