lance-format/lance

Support Arrow Tensor type

Open

#108 创建于 2022年8月15日

在 GitHub 查看
 (3 评论) (0 反应) (1 负责人)Rust (695 fork)github user discovery
A-pythonarrowenhancementhelp wantedrust

仓库指标

Star
 (6,582 star)
PR 合并指标
 (平均合并 6天 1小时) (30 天内合并 219 个 PR)

描述

edit: use arrow::extension::FixedShapeTensorType

https://arrow.apache.org/docs/dev/format/CanonicalExtensions.html#fixed-shape-tensor-extension

Problem

In arrow, Tensors do not have corresponding Array type, as it can not be used to store in a Table / Dataset / RecordBatch.

To support ML datasets, it is desirable to have tensors stored within the datasets.

Desired Behavior

To support parametrized Tensors in a Table / Datasets. A TensorArray and TensorType should allow parameters, i.e., shade, data type (i.e., bid_width) and etc.

class TensorType {
   TensorType(data_type, shade, ...)
}

贡献者指南