Databricks SQL ExecuteStatement Failing During Chunked Updates to a Single Cell (Large JSON Field)

未關閉

#592 建立於 2025年6月10日 0 則留言 0 個 reaction 已指派 0 人在 GitHub 檢視

還沒有人認領這個 Issue。

評估

難度: 5/5
預估耗時: 一週以上
新手友好度: 25/100

Issue 類型: 缺陷
描述清晰度: 需要釐清
活躍度: 停滯
技術堆疊: python, sql, sqlalchemy
領域: database

研究方向

從報告中描述的 SQLAlchemy 和 Databricks SQL ExecuteStatement 路徑開始，然後使用大型 JSON 欄位重現分塊 UPDATE 序列。比較失敗時的請求大小，並檢查 connector 對重試代碼 429 和 503 的處理。完成的標準是記錄請求限制和一種獲批准的增量更新實務。

由索引模型根據 Issue 內容生成。

描述

We’re encountering issues with Databricks SQL when attempting to update a row that contains a large JSON array field. Originally, we tried inserting the entire JSON directly into the column, but this failed due to request size limitations.
To address that, we redesigned the approach to:

Split the full JSON into smaller chunks (~100 items)
Append each chunk incrementally to the same row/column using separate UPDATE statements
Commit each chunk using a new thread and session to ensure SQLAlchemy thread safety

Despite chunking, the request eventually fails when the dmt_data field grows large enough (presumably ~1–2MB compressed). The SQL API returns:

pgsqlCopyEdit(databricks.sql.exc.RequestError) Error during request to server. 
ExecuteStatement command can only be retried for codes 429 and 503

This confirms that each UPDATE's request body is still exceeding Databricks SQL's internal limits, even though we’re only appending small pieces.
What We’re Looking For:

Confirmation of the exact request body size limit for INSERT/UPDATE operations over Databricks SQL
Recommended practice for incrementally updating a single JSON column that grows over time

主要語言: Python
星號: 233
分支: 152
平均合併: 21 小時 5 分鐘
30 天內合併 PR: 10