Unknown Error - Getting from Databricks SQL Python - From PyArrow module (pyarrow.lib.ArrowException)

Ouverte
  • Quelque chose ne va pas ?
#501 2 commentaires 0 réactions 1 personne assignée Voir sur GitHub

Les mainteneurs répondent en général sous 1 jour

@samikshya-db y travaille déjà.

Depuis le 25/2/2025.

Évaluation

Cette issue n'a pas encore été évaluée.

Description

Versions:
Python: 3.11.9
Databricks SQL Connector: 3.3.0
pandas: 2.1.4
PyArrow: 16.1.0

Code:
Below is the code I'm using to fetch data from databricks which has some utf-8 encoded char (�) in the data.

sql_query = f""" SELECT Column FROM table LIMIT 10 """

host = os.getenv("DATABRICKS_HOST")
http_path = os.getenv("DATABRICKS_HTTP_PATH")

connection = sql.connect(server_hostname=host, http_path=http_path)
with conn.cursor() as cursor:
        cursor.execute(sql_query)
        response = fetchmany(cursor)

Error:

    response = fetchmany(cursor)
               ^^^^^^^^^^^^^^^^^
  File "/usr/local/airflow/.local/lib/python3.11/site-packages/my_agent/app_helper/db_helper.py", line 35, in fetchmany
    query_results = cursor.fetchmany(fetch_size)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/airflow/.local/lib/python3.11/site-packages/my_agent_connectors/dq_databricks/dependencies/databricks/sql/client.py", line 976, in fetchmany
    return self.active_result_set.fetchmany(size)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/airflow/.local/lib/python3.11/site-packages/my_agent_connectors/dq_databricks/dependencies/databricks/sql/client.py", line 1235, in fetchmany
    return self._convert_arrow_table(self.fetchmany_arrow(size))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/airflow/.local/lib/python3.11/site-packages/my_agent_connectors/dq_databricks/dependencies/databricks/sql/client.py", line 1161, in _convert_arrow_table
    df = table_renamed.to_pandas(
         ^^^^^^^^^^^^^^^^^^^^^^^^
  File "pyarrow/array.pxi", line 884, in pyarrow.lib._PandasConvertible.to_pandas
  File "pyarrow/table.pxi", line 4192, in pyarrow.lib.Table._to_pandas
  File "/usr/local/airflow/.local/lib/python3.11/site-packages/pyarrow/pandas_compat.py", line 776, in table_to_dataframe
    blocks = _table_to_blocks(options, table, categories, ext_columns_dtypes)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/airflow/.local/lib/python3.11/site-packages/pyarrow/pandas_compat.py", line 1131, in _table_to_blocks
    return [_reconstruct_block(item, columns, extension_columns)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/airflow/.local/lib/python3.11/site-packages/pyarrow/pandas_compat.py", line 1131, in <listcomp>
    return [_reconstruct_block(item, columns, extension_columns)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/airflow/.local/lib/python3.11/site-packages/pyarrow/pandas_compat.py", line 736, in _reconstruct_block
    pd_ext_arr = pandas_dtype.__from_arrow__(arr)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/airflow/.local/lib/python3.11/site-packages/pandas/core/arrays/string_.py", line 230, in __from_arrow__
    arr = arr.to_numpy(zero_copy_only=False)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "pyarrow/array.pxi", line 1581, in pyarrow.lib.Array.to_numpy
  File "pyarrow/error.pxi", line 91, in pyarrow.lib.check_status
pyarrow.lib.ArrowException: Unknown error: Wrapping Ruta 204 � Zapote failed```
Langage dominant
Python
Étoiles
236
Forks
162
Merge moyen
13 h 13 min
PR mergées (30 j)
23

Préparer son environnement

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Autres issues de databricks/databricks-sql-python

Toutes les issues de databricks/databricks-sql-python

Issues similaires

Plus d'issues Python

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.