Request: first-class representation of file-backed datasets (e.g. Parquet) in Dataset.source (or adjacent fields)

Abierto
#104 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
30/100
Tipo de issue
Nueva funcionalidad
Claridad
Necesita aclaración
Estado de actividad
Tranquilo
Stack tecnológico
sql

Línea de trabajo

El issue no menciona archivos, pruebas ni puntos de entrada de implementación. Empieza revisando las definiciones actuales de Dataset.source y custom_extensions y, después, examina las tres direcciones propuestas y cualquier debate previo del grupo de trabajo; el trabajo estaría terminado cuando exista una representación acordada o un perfil documentado para datasets respaldados por archivos.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Summary
Today the OSI core spec describes Dataset.source as a reference to an underlying physical table/view (e.g. database.schema.table) or query. That model fits warehouse-centric semantics well but does not describe file- or object-store-backed relations (Parquet, directories/globs, file: / s3: / gs: URIs, etc.), which are common in analytical and lakehouse workflows.

This issue asks whether the working group has already discussed file-backed / columnar-file sources, and if so, what direction was considered. If not, we would like to explore first-class (core or well-documented optional profile) support so documents are portable without overloading source or relying only on opaque custom_extensions.

Motivation / use cases

  • Local development and tests: point a semantic dataset at Parquet on disk without a catalog.
  • Object storage: datasets backed by s3://, gs://, or HTTPS without a SQL warehouse name.
  • Interoperability: engines that compile OSI to Substrait / SQL often need a stable logical name for the dataset while physical scan targets are resolved separately; the spec could make that split explicit.
  • Consistency: avoids each implementation inventing incompatible conventions inside COMMON custom extensions.

Current spec (as we read it)

  • Core source is a single string, with examples in database.schema.table form.
  • Extensibility exists via custom_extensions / Vendor, which works but does not give a shared, validated shape for “this dataset is Parquet at these locations.”

Questions for the working group

  1. Has file-backed / columnar-file (especially Parquet) been discussed? If yes, pointers to notes or prior issues would be appreciated.
  2. Is the intent that source remain strictly warehouse-style, with all file semantics in extensions, or is there appetite to extend the core model?
  3. Would the WG prefer:
    A) Extending Dataset with optional structured fields (e.g. physical / storage with kind, locations[], format), while keeping source as the logical relation id for queries; or
    B) A documented optional profile (separate schema or appendix) for “file datasets” without changing minimal core; or
    C) Strengthening guidance + JSON schema for a specific COMMON (or new vendor) extension payload so all implementations converge.

Sketch of a possible shape (for discussion only)
Not a proposal to adopt verbatim, only to illustrate the gap:

  • name: logical dataset id (unchanged).
  • source: stable logical relation name used in generated SQL/Substrait (e.g. qualified name), or explicit split between “logical id” and “physical” if the WG wants that separation.
  • Optional storage descriptor: e.g. format parquet, one or more URIs (files, globs, prefixes), optional partition / read hints.

Happy to help refine this with implementer feedback (e.g. from tools that emit OSI and target Arrow/DataFusion/Iceberg).

Lenguaje dominante
Python
Estrellas
2.7k
Forks
353
Merge medio
3 d 19 h
PR fusionados (30 d)
86

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de apache/ossie

Todos los issues de apache/ossie

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.