How to insert into VARIANT column?

未關閉
  • 哪裡不對？
#681 1 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視

維護者通常 1 天內回覆

還沒有人認領這個 Issue。

評估

難度
2/5
預估耗時
1-3 小時
新手友好度
35/100
Issue 類型
文件
描述清晰度
基本清楚
活躍度
停滯
技術堆疊
python, sql
領域
databases, documentation

研究方向

首先，使用 executemany、JSON 字串和用於 VARIANT 欄位的 dict，重現 Python databricks-sql-connector 4.0.5 範例。記錄支援哪種參數形式，以及已完成的插入對於 schema_of_variant(content) 應顯示什麼，包括相關的 SQL 行為。

由索引模型根據 Issue 內容生成。

描述

As we all know, Databricks has a VARIANT data type that is more performant than standard JSON strings for nested data. However, I don't understand how I am supposed to insert into such a column using databricks-sql-connector==4.0.5 though.

If you want to write to a PostgreSQL JSON column in psycopg2, you use json dump. This trick did not seem to work here. I also tried custom escaped JSON and simply dict, but also no.

import databricks.sql
import json

# Replace with your Databricks SQL warehouse details
server_hostname = "<SERVER_HOSTNAME>"
http_path = "<HTTP_PATH>"
access_token = "<ACCESS_TOKEN>"

# Example data to insert
content = {
    "age": 29,
    "city": "New York"
}
data = [
    (1, "Alice", json.dumps(content)),
    2, "Bob", None), # (2, "Bob", data) # Raises error
    (3, "Charlie", """{"age": 29, "city": "New York"}"""),
]

# Connect to Databricks SQL
with databricks.sql.connect(
    server_hostname=server_hostname,
    http_path=http_path,
    access_token=access_token
) as connection:
    with connection.cursor() as cursor:
        # Insert data into Delta table
        cursor.execute("CREATE OR REPLACE TABLE test_catalog.ad_hoc.variant_test (id INT, name STRING, content VARIANT) USING DELTA")
        cursor.executemany(
            "INSERT INTO prod_catalog.ad_hoc.variant_test (id, name, content) VALUES (?, ?, ?)",
            data
        )

If I query the resulting table,

SELECT *, content:age, parse_json("{\"age\": 29, \"city\": \"New York\"}") FROM ad_hoc.variant_test ORDER BY id;

1 and 3 succeeds, but it seems to end up as a "flat"STRING (schema_of_variant(content)) instead of OBJECT<age: BIGINT, city: STRING>.

Image

Attempt 2 (insert a dict) raises the error

databricks.sql.exc.ServerOperationError: [DATATYPE_MISMATCH.CAST_WITHOUT_SUGGESTION] Cannot resolve "content" due to data type mismatch: cannot cast "MAP<VOID, VOID>" to "VARIANT". SQLSTATE: 42K09; line 1 pos 0

Could we add documentation for how this is supposed to work? I guess MAP, STRUCT and VARIANT all map to Python type dict in some sense. It is only that VARIANT don't have a predefined schema like MAP or STRUCT, and unlike MAP can be nested?

主要語言
Python
星號
236
分支
162
平均合併
14 小時 22 分鐘
30 天內合併 PR
25

環境準備

從這裡開始

  1. 先讀完整個 Issue，再讀專案的貢獻指南。
  2. 在 Issue 下留言說明你要接手 —— 這能避免兩個人做同樣的事。
  3. Fork 儲存庫，在一個分支上完成修改。
  4. 送出 Pull Request，並在描述裡引用這個 Issue 編號。

databricks/databricks-sql-python 的其他 Issue

查看 databricks/databricks-sql-python 的全部 Issue

相似的 Issue

更多 Python Issue

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。