Unknown Error - Getting from Databricks SQL Python - From PyArrow module (pyarrow.lib.ArrowException)
@samikshya-db 已經在處理了。
開始於 2025年2月25日。
評估
這個 Issue 還沒有評估資料。
描述
Versions:
Python: 3.11.9
Databricks SQL Connector: 3.3.0
pandas: 2.1.4
PyArrow: 16.1.0
Code:
Below is the code I'm using to fetch data from databricks which has some utf-8 encoded char (�) in the data.
sql_query = f""" SELECT Column FROM table LIMIT 10 """
host = os.getenv("DATABRICKS_HOST")
http_path = os.getenv("DATABRICKS_HTTP_PATH")
connection = sql.connect(server_hostname=host, http_path=http_path)
with conn.cursor() as cursor:
cursor.execute(sql_query)
response = fetchmany(cursor)
Error:
response = fetchmany(cursor)
^^^^^^^^^^^^^^^^^
File "/usr/local/airflow/.local/lib/python3.11/site-packages/my_agent/app_helper/db_helper.py", line 35, in fetchmany
query_results = cursor.fetchmany(fetch_size)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/airflow/.local/lib/python3.11/site-packages/my_agent_connectors/dq_databricks/dependencies/databricks/sql/client.py", line 976, in fetchmany
return self.active_result_set.fetchmany(size)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/airflow/.local/lib/python3.11/site-packages/my_agent_connectors/dq_databricks/dependencies/databricks/sql/client.py", line 1235, in fetchmany
return self._convert_arrow_table(self.fetchmany_arrow(size))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/airflow/.local/lib/python3.11/site-packages/my_agent_connectors/dq_databricks/dependencies/databricks/sql/client.py", line 1161, in _convert_arrow_table
df = table_renamed.to_pandas(
^^^^^^^^^^^^^^^^^^^^^^^^
File "pyarrow/array.pxi", line 884, in pyarrow.lib._PandasConvertible.to_pandas
File "pyarrow/table.pxi", line 4192, in pyarrow.lib.Table._to_pandas
File "/usr/local/airflow/.local/lib/python3.11/site-packages/pyarrow/pandas_compat.py", line 776, in table_to_dataframe
blocks = _table_to_blocks(options, table, categories, ext_columns_dtypes)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/airflow/.local/lib/python3.11/site-packages/pyarrow/pandas_compat.py", line 1131, in _table_to_blocks
return [_reconstruct_block(item, columns, extension_columns)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/airflow/.local/lib/python3.11/site-packages/pyarrow/pandas_compat.py", line 1131, in <listcomp>
return [_reconstruct_block(item, columns, extension_columns)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/airflow/.local/lib/python3.11/site-packages/pyarrow/pandas_compat.py", line 736, in _reconstruct_block
pd_ext_arr = pandas_dtype.__from_arrow__(arr)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/airflow/.local/lib/python3.11/site-packages/pandas/core/arrays/string_.py", line 230, in __from_arrow__
arr = arr.to_numpy(zero_copy_only=False)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "pyarrow/array.pxi", line 1581, in pyarrow.lib.Array.to_numpy
File "pyarrow/error.pxi", line 91, in pyarrow.lib.check_status
pyarrow.lib.ArrowException: Unknown error: Wrapping Ruta 204 � Zapote failed```
- 主要語言
- Python
- 星號
- 233
- 分支
- 152
- 平均合併
- 21 小時 5 分鐘
- 30 天內合併 PR
- 10
貢獻指南
從這裡開始
- 先讀完整個 Issue,再讀專案的貢獻指南。
- 在 Issue 下留言說明你要接手 —— 這能避免兩個人做同樣的事。
- Fork 儲存庫,在一個分支上完成修改。
- 送出 Pull Request,並在描述裡引用這個 Issue 編號。
databricks/databricks-sql-python 的其他 Issue
-
難度 2/5 1-3 小時 新手友好度 78/100
-
難度 2/5 1-3 小時 新手友好度 76/100
-
難度 2/5 1-3 小時 新手友好度 78/100
-
難度 2/5 1-3 小時 新手友好度 72/100
-
難度 2/5 1-3 小時 新手友好度 84/100
查看 databricks/databricks-sql-python 的全部 Issue
相似的 Issue
-
enhancement
難度 2/5 1-3 小時 新手友好度 70/100
canonical/paas-charm#368 · 1 則留言 ·
-
難度 2/5 1-3 小時 新手友好度 75/100
-
tech debt
難度 2/5 1-3 小時 新手友好度 75/100
-
難度 1/5 1 小時以內 新手友好度 90/100
StevenBlack/hosts#3256 ·
-
難度 1/5 1 小時以內 新手友好度 90/100
qualcomm/qai-appbuilder#275 ·