Upgrade Databricks SQL Alchemy version to avoid issues with metadata operations

Open Beginner friendly
#38,658 5 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
70/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
python, sqlalchemy
Domain
databases

Research direction

Find the dependency pin for the Databricks SQLAlchemy connector and trace the metadata-discovery path used for table browsing. Verify the change against catalogs and schemas containing hyphens; done means the dependency is updated to a version that generates correctly quoted SHOW TABLES queries without breaking existing metadata operations.

Written by the indexing model from the issue text.

Description

data:connect:databricks dependencies:python validation:required
Bug description

Hi there,

Some of the users are hitting an issue on Superset v6 using the databricks python connector

The metadata discovery (table browser) is currently broken when using catalogs and/or schemas containing hyphens (which is very common in Databricks Unity Catalog). Superset fails to fetch the table list because the generated SQL is not properly quoted.

For example, it currently generates:

SHOW TABLES FROM my-staging-catalog.my-poc-schema

Instead of:

SHOW TABLES FROM `my-staging-catalog`.`my-poc-schema`
This triggers a [INVALID_IDENTIFIER] error because the hyphen is interpreted as a subtraction operator rather than part of the identifier.

[INVALID_IDENTIFIER] The unquoted identifier my-staging-catalog is invalid and must be back quoted as: `my-staging-catalog`.
Unquoted identifiers can only contain ASCII letters ('a' - 'z', 'A' - 'Z'), digits ('0' - '9'), and underbar ('_').

This was fixed in SQL Alchemy in v2.0.4 but the repo has pinned v1.0.5
Can we please update to point to the recent version of Databricks SQL Alchemy?

Screenshots/recordings

No response

Superset version

master / latest-dev

Python version

3.9

Node version

Not applicable

Browser

Chrome

Additional context

No response

Checklist
  • I have searched Superset docs and Slack and didn't find a solution to my problem.
  • I have searched the GitHub issue tracker and didn't find a similar bug report.
  • I have checked Superset's logs for errors and if I found a relevant Python stacktrace, I included it here as text in the "additional context" section.
Dominant language
Python
Stars
74.8k
Forks
18.4k
Avg merge
2d 8h
Merged PRs (30d)
692

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from apache/superset

All issues in apache/superset

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.