Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

Polars database reader/writer record file metadata, so SQL lineage is missing

Aperta
  • Qualcosa non va?
#1,740 2 commenti 0 reazioni 1 assegnatario Vedi su GitHub

I maintainer di solito rispondono entro 2 giorni

@Dev-iL ci sta già lavorando.

Dal 27/9/2026.

  • #1743 di @Dev-iL — aperta

Valutazione

Questa issue non è ancora stata valutata.

Descrizione

Related: #1720

PolarsDatabaseReader and PolarsDatabaseWriter (in both polars_post_1_0_0_extensions.py and polars_pre_1_0_0_extension.py) describe database I/O with get_file_and_dataframe_metadata. They pass the SQL query or table name in where a file path is expected:

# PolarsDatabaseReader.load_data
metadata = utils.get_file_and_dataframe_metadata(self.query, df)
# PolarsDatabaseWriter.save_data
return utils.get_file_and_dataframe_metadata(self.table_name, data)

This happens whatever connection is passed, including a SQLAlchemy Engine.

Effects

  • The metadata has no sql_metadata block. So the OpenLineageAdapter can't identify the tables, and sql_dataset_identity="datasource" (from #1720) resolves nothing: sql_datasets() returns SQL datasource is unknown; no dataset emitted.
  • Lineage treats database I/O as filesystem I/O:
    • a read becomes an input dataset named after the node (e.g. orders_df), with storageLayer="FileSystem" and the SQL text as its dataSource URI;
    • a write becomes an output dataset named after the table, also marked FileSystem;
    • no sql job facet is emitted.
  • file_metadata carries meaningless values, e.g. path: "SELECT * FROM orders", scheme: "", and a note that the file does not exist.

Reproduction

import polars as pl
from sqlalchemy import create_engine
from hamilton.plugins.polars_post_1_0_0_extensions import PolarsDatabaseReader

engine = create_engine("sqlite:///a.db")  # with a table `orders`
df, md = PolarsDatabaseReader(query="SELECT * FROM orders", connection=engine).load_data(pl.DataFrame)
print(sorted(md))                  # ['dataframe_metadata', 'file_metadata']
print(md["file_metadata"]["path"])  # 'SELECT * FROM orders'

Proposed fix

  1. In both Polars database classes, record get_sql_metadata(query_or_table, results, db_connection=self.connection, operation="read" | "write") plus get_dataframe_metadata, as PandasSqlReader and PandasSqlWriter do. SQLAlchemy URLs, Engines and Connections, and sqlite3 connections, then get full datasource identity.
  2. Extend get_sql_source to recognise the other connections Polars accepts (ADBC connections, connectorx URI strings, cursors). Until then these are left out of lineage with a note, which is safe but gives no lineage.
  3. Add Polars read and write tests against SQLite, mirroring the pandas SQL lineage tests.

Compatibility

Metadata for Polars database I/O changes from file_metadata to sql_metadata. Default-mode ("legacy") OpenLineage output changes with it: the fake filesystem datasets are replaced by the legacy SQL datasets. Anyone consuming file_metadata from these classes would be affected, so this probably deserves a changelog entry.

Lingua principale
Jupyter Notebook
Stelle
2.6k
Fork
218
Merge medio
2g 1m
PR unite (30g)
17

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di apache/hamilton

Tutte le issue di apache/hamilton

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.