Request: first-class representation of file-backed datasets (e.g. Parquet) in Dataset.source (or adjacent fields)

Open
#104 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
30/100
Issue type
Feature
Clarity
Needs clarification
Activity status
Quiet
Tech stack
sql

Research direction

The issue names no files, tests, or implementation entry points. Start by reviewing the current Dataset.source and custom_extensions definitions, then examine the three proposed directions and any prior working-group discussion; done would require an agreed representation or documented profile for file-backed datasets.

Written by the indexing model from the issue text.

Description

Summary
Today the OSI core spec describes Dataset.source as a reference to an underlying physical table/view (e.g. database.schema.table) or query. That model fits warehouse-centric semantics well but does not describe file- or object-store-backed relations (Parquet, directories/globs, file: / s3: / gs: URIs, etc.), which are common in analytical and lakehouse workflows.

This issue asks whether the working group has already discussed file-backed / columnar-file sources, and if so, what direction was considered. If not, we would like to explore first-class (core or well-documented optional profile) support so documents are portable without overloading source or relying only on opaque custom_extensions.

Motivation / use cases

  • Local development and tests: point a semantic dataset at Parquet on disk without a catalog.
  • Object storage: datasets backed by s3://, gs://, or HTTPS without a SQL warehouse name.
  • Interoperability: engines that compile OSI to Substrait / SQL often need a stable logical name for the dataset while physical scan targets are resolved separately; the spec could make that split explicit.
  • Consistency: avoids each implementation inventing incompatible conventions inside COMMON custom extensions.

Current spec (as we read it)

  • Core source is a single string, with examples in database.schema.table form.
  • Extensibility exists via custom_extensions / Vendor, which works but does not give a shared, validated shape for “this dataset is Parquet at these locations.”

Questions for the working group

  1. Has file-backed / columnar-file (especially Parquet) been discussed? If yes, pointers to notes or prior issues would be appreciated.
  2. Is the intent that source remain strictly warehouse-style, with all file semantics in extensions, or is there appetite to extend the core model?
  3. Would the WG prefer:
    A) Extending Dataset with optional structured fields (e.g. physical / storage with kind, locations[], format), while keeping source as the logical relation id for queries; or
    B) A documented optional profile (separate schema or appendix) for “file datasets” without changing minimal core; or
    C) Strengthening guidance + JSON schema for a specific COMMON (or new vendor) extension payload so all implementations converge.

Sketch of a possible shape (for discussion only)
Not a proposal to adopt verbatim, only to illustrate the gap:

  • name: logical dataset id (unchanged).
  • source: stable logical relation name used in generated SQL/Substrait (e.g. qualified name), or explicit split between “logical id” and “physical” if the WG wants that separation.
  • Optional storage descriptor: e.g. format parquet, one or more URIs (files, globs, prefixes), optional partition / read hints.

Happy to help refine this with implementer feedback (e.g. from tools that emit OSI and target Arrow/DataFusion/Iceberg).

Dominant language
Python
Stars
2.7k
Forks
353
Avg merge
3d 19h
Merged PRs (30d)
86

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from apache/ossie

All issues in apache/ossie

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.