Databricks SQL ExecuteStatement Failing During Chunked Updates to a Single Cell (Large JSON Field)
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 25/100
Research direction
Start with the SQLAlchemy and Databricks SQL ExecuteStatement path described in the report, then reproduce the chunked UPDATE sequence using a large JSON field. Compare the request size at failure and inspect the connector’s handling of retry codes 429 and 503. Done means documenting the request limit and an approved incremental-update practice.
Written by the indexing model from the issue text.
Description
We’re encountering issues with Databricks SQL when attempting to update a row that contains a large JSON array field. Originally, we tried inserting the entire JSON directly into the column, but this failed due to request size limitations.
To address that, we redesigned the approach to:
- Split the full JSON into smaller chunks (~100 items)
- Append each chunk incrementally to the same row/column using separate UPDATE statements
- Commit each chunk using a new thread and session to ensure SQLAlchemy thread safety
Despite chunking, the request eventually fails when the dmt_data field grows large enough (presumably ~1–2MB compressed). The SQL API returns:
pgsqlCopyEdit(databricks.sql.exc.RequestError) Error during request to server.
ExecuteStatement command can only be retried for codes 429 and 503
This confirms that each UPDATE's request body is still exceeding Databricks SQL's internal limits, even though we’re only appending small pieces.
What We’re Looking For:
- Confirmation of the exact request body size limit for INSERT/UPDATE operations over Databricks SQL
- Recommended practice for incrementally updating a single JSON column that grows over time
- Dominant language
- Python
- Stars
- 233
- Forks
- 152
- Avg merge
- 1d 23h
- Merged PRs (30d)
- 11
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from databricks/databricks-sql-python
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
-
engineer-bot
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
databricks/databricks-sql-python#860 · 3 comments ·
All issues in databricks/databricks-sql-python
Similar issues
-
agent-ready documentation needs-triage
Difficulty 1/5 1-3 hours Newbie friendliness 88/100
-
documentation
Difficulty 1/5 Under an hour Newbie friendliness 91/100
-
workflow-status page template still says reusable workflows are "triggered only by workflow_call:" Open
Difficulty 1/5 Under an hour Newbie friendliness 92/100
-
instance instance add
Difficulty 1/5 Under an hour Newbie friendliness 72/100
searxng/searx-instances#939 · 1 comment ·
-
area-deployment area-integrations triage:bot-seen
Difficulty 2/5 Half a day Newbie friendliness 86/100