DB engine for pandas: sql.connect or sqlalchemy
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 25/100
Research direction
The issue compares databricks.sql.connect with SQLAlchemy and pandas.read_sql, including CloudFetch and possible serialization overhead. Start by reviewing the linked closed pandas-related pull requests and the pandas.read_sql integration. Done means documenting the recommended API, performance implications, and whether both paths retain CloudFetch.
Written by the indexing model from the issue text.
Description
Hello,
I was wondering what's the best practice for using this package with pandas.
- It's possible to create a
databricks.sql.connectand pass it topandas.read_sql. This works however it raises
UserWarning: pandas only supports SQLAlchemy connectable (engine/connection) or database string URI or sqlite3 DBAPI2
connection. Other DBAPI2 objects are not tested. Please consider using SQLAlchemy.
- Alternatively it's possible to use SQLAlchemy with a
databricks://URL and pass that to pandas. Doesn't it mean an extra serialization step performance wise though?
What's the recommended way, in particular regarding performance? Would both use CloudFetch for larger queries? I see there are some fixes/improvements done for pandas done in PRs so which API should be used to benefit from those?
Thanks!
cc @kravets-levko
- Dominant language
- Python
- Stars
- 233
- Forks
- 152
- Avg merge
- 21h 5m
- Merged PRs (30d)
- 10
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from databricks/databricks-sql-python
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
All issues in databricks/databricks-sql-python
Similar issues
-
essnmx good first issue
Difficulty 1/5 Under an hour Newbie friendliness 95/100
-
[Feature] 奇物选择添加优先级 Open
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
syfoud/Simulated_Scepter#174 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
Giskard-AI/giskard-oss#2840 · 1 comment ·
-
A claim comment carrying the issue number is silently declined while the workflow reports success Openarea: repo bug perceived difficulty: 2
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
yeti-platform/yeti#1380 ·