Cross catalog query support

Open
#49 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
30/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Stale
Tech stack
python, sqlalchemy
Domain
databases

Research direction

Start by reviewing the documented connection URL with schema and catalog, then compare it with the table_args and quoted_name examples in the issue. Define what per-table catalog support should mean for cross-schema and cross-catalog queries, and confirm that the resulting table declarations can join the described Delta and LakeBase tables through one engine.

Written by the indexing model from the issue text.

Description

I would like to use databricks-sqlalchemy to query tables located in different schemas and catalogs (or even JOIN them) using the same engine, current documentation presents only a single configuration where both schema and catalog are provided as part of the URL:
databricks://token:{token}@{hostname}?http_path={http_path}&schema={schema}&catalog={catalog}

I discovered (it's not documented) that i can remove both parameters from the URL and use __table_args__ to provide these
for every Table:

class SomeTable(Base):
    __tablename__ = "some_table"
    __table_args__ = (
        {"schema": quoted_name(f"my_catalog.my_schema", quote=False)},
    )

    some_field = Column(String(6), nullable=False)

It looks like a hack, because the schema property value is actually a catalog + schema, and i need to use quoted_name to avoid some errors.

Would it be possible to support some dialect specific property, for example databricks_catalog to define a catalog as part of the __table_args__? It would look like this:

__table_args__ = (
    {"schema": "my_schema", "databricks_catalog": "my_catalog"},
)

Just catalog is not allowed here because it leads to an error:
TypeError: Additional arguments should be named <dialectname>_<argument>, got 'catalog'
But dialect specific should be fine.

Having such feature would be easier for us to use the library with our use case (we want to join Delta tables with LakeBase ones added to Unity Catalog)

Thanks

Dominant language
Python
Stars
24
Forks
18
No merged PRs in 30d

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from databricks/databricks-sqlalchemy

All issues in databricks/databricks-sqlalchemy

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.