Request: first-class representation of file-backed datasets (e.g. Parquet) in Dataset.source (or adjacent fields)
维护者通常 1 天内回复
还没有人认领这个 Issue。
评估
- 难度
- 5/5
- 预计耗时
- 一周以上
- 新手友好度
- 30/100
- Issue 类型
- 功能
- 描述清晰度
- 需要澄清
- 活跃度
- 冷清
- 技术栈
- sql
调研方向
该 issue 未指出文件、测试或实现入口。首先查看当前的 Dataset.source 和 custom_extensions 定义,然后检查提出的三个方向以及之前的任何工作组讨论;对于文件后端 dataset,达成一种共识表示形式或记录完善的 profile 后即可视为完成。
由索引模型根据 Issue 内容生成。
描述
Summary
Today the OSI core spec describes Dataset.source as a reference to an underlying physical table/view (e.g. database.schema.table) or query. That model fits warehouse-centric semantics well but does not describe file- or object-store-backed relations (Parquet, directories/globs, file: / s3: / gs: URIs, etc.), which are common in analytical and lakehouse workflows.
This issue asks whether the working group has already discussed file-backed / columnar-file sources, and if so, what direction was considered. If not, we would like to explore first-class (core or well-documented optional profile) support so documents are portable without overloading source or relying only on opaque custom_extensions.
Motivation / use cases
- Local development and tests: point a semantic dataset at Parquet on disk without a catalog.
- Object storage: datasets backed by s3://, gs://, or HTTPS without a SQL warehouse name.
- Interoperability: engines that compile OSI to Substrait / SQL often need a stable logical name for the dataset while physical scan targets are resolved separately; the spec could make that split explicit.
- Consistency: avoids each implementation inventing incompatible conventions inside COMMON custom extensions.
Current spec (as we read it)
- Core source is a single string, with examples in database.schema.table form.
- Extensibility exists via custom_extensions / Vendor, which works but does not give a shared, validated shape for “this dataset is Parquet at these locations.”
Questions for the working group
- Has file-backed / columnar-file (especially Parquet) been discussed? If yes, pointers to notes or prior issues would be appreciated.
- Is the intent that source remain strictly warehouse-style, with all file semantics in extensions, or is there appetite to extend the core model?
- Would the WG prefer:
A) Extending Dataset with optional structured fields (e.g. physical / storage with kind, locations[], format), while keeping source as the logical relation id for queries; or
B) A documented optional profile (separate schema or appendix) for “file datasets” without changing minimal core; or
C) Strengthening guidance + JSON schema for a specific COMMON (or new vendor) extension payload so all implementations converge.
Sketch of a possible shape (for discussion only)
Not a proposal to adopt verbatim, only to illustrate the gap:
- name: logical dataset id (unchanged).
- source: stable logical relation name used in generated SQL/Substrait (e.g. qualified name), or explicit split between “logical id” and “physical” if the WG wants that separation.
- Optional storage descriptor: e.g. format parquet, one or more URIs (files, globs, prefixes), optional partition / read hints.
Happy to help refine this with implementer feedback (e.g. from tools that emit OSI and target Arrow/DataFusion/Iceberg).
- 主要语言
- Python
- 星标
- 2.7k
- 派生
- 353
- 平均合并
- 3 天 19 小时
- 30 天内合并 PR
- 86
环境准备
- 没有 Dockerfile 或 Docker Compose 文件
- 有 Pull Request 模板
- 阅读贡献指南
从这里开始
- 先读完整个 Issue,再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库,在一个分支上完成修改。
- 提交 Pull Request,并在描述里引用这个 Issue 编号。
apache/ossie 的其他 Issue
-
needs-triage
难度 2/5 1-3 小时 新手友好度 78/100
维护者通常 1 天内回复
-
needs-triage
难度 2/5 1-3 小时 新手友好度 52/100
维护者通常 1 天内回复
-
validation: field reference resolution is case-insensitive but name uniqueness is case-sensitive可能已有人在做 @MikeNitsenko 于 6 天前认领。 未关闭needs-triage
难度 5/5 一周以上 新手友好度 25/100
维护者通常 1 天内回复
-
needs-triage
难度 5/5 一周以上 新手友好度 25/100
维护者通常 1 天内回复
-
needs-triage
难度 5/5 一周以上 新手友好度 25/100
维护者通常 1 天内回复
相似的 Issue
-
难度 1/5 1 小时以内 新手友好度 86/100
sandbox-quantum/switch#2163 ·
维护者通常 1 天内回复
-
bug pydanty:is-working
难度 2/5 1-3 小时 新手友好度 82/100
pydantic/pydantic-ai#13939 ·
维护者通常 1 天内回复
-
priority: low python
难度 2/5 1-3 小时 新手友好度 84/100
LearningCircuit/local-deep-research#8849 ·
维护者通常 1 天内回复
-
难度 1/5 1 小时以内 新手友好度 86/100
francescopace/espectre#223 ·
维护者通常 3 天内回复
-
good first issue Status: Triaged
难度 1/5 1 小时以内 新手友好度 84/100
canonical/imagecraft#538 · 1 条评论 ·
维护者通常 2 天内回复