ExecuteMany performance is insanely bad
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 20/100
- Issue type
- Feature
- Clarity
- Needs clarification
- Activity status
- Stale
- Domain
- data-engineering, databases
Research direction
Start by locating the connector's executemany implementation and reading the documented behavior that describes it as a naive loop. Investigate how a single SQL statement using VALUES(...) could support the requested batching for writes to a Delta table, and define completion around improved performance for non-trivial dataframe loads.
Written by the indexing model from the issue text.
Description
I know that the documentation makes it clear that executemany is a naive for loop:
No optimizations of the query (like batching) will be performed.
But it's 2025 and please have a more optimized executemany that just issues a single SQL statement using VALUES(...) or something so this is actually usable in a data pipeline. Otherwise, trying to use the databricks SQL connector to write any non-trivial dataframe to a delta table is pointless.
Thanks!
- Dominant language
- Python
- Stars
- 236
- Forks
- 162
- Avg merge
- 13h 13m
- Merged PRs (30d)
- 23
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from databricks/databricks-sql-python
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
databricks/databricks-sql-python#1034 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
databricks/databricks-sql-python#1012 ·
Maintainers usually reply within 1 day
-
Token federation: `_exchange_token` raises KeyError('access_token') on error responses, discarding the real failure reasonPossibly taken @TangoEnSkai claimed this 81 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
databricks/databricks-sql-python#904 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
databricks/databricks-sql-python#865 ·
Maintainers usually reply within 1 day
-
engineer-bot
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
databricks/databricks-sql-python#860 · 3 comments ·
Maintainers usually reply within 1 day
All issues in databricks/databricks-sql-python
Similar issues
-
Difficulty 1/5 1-3 hours Newbie friendliness 88/100
aws-samples/sample-per-model-bedrock#857 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 70/100
521xueweihan/HelloGitHub#4328 ·
-
documentation maintainer-managed
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
awslabs/visual-asset-management-system#1418 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day