In `1.10`, adapter.get_columns_in_relation switched to a new query causing some old logic to fail (that depended on the datatype output)

Aperta
  • Qualcosa non va?
#1,148 7 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 6 giorni

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
3/5
Tempo stimato
1-2 giorni
Idoneità per principianti
48/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Tranquilla
Stack tecnologico
python, sql
Ambito
database

Direzione di ricerca

Inizia dal punto di ingresso adapter.get_columns_in_relation() e confronta le differenze tra le query dei metadati e i tipi analizzati di 1.9 e 1.10. Esegui il modello dbt fornito e verifica la riproduzione delle macro per i tipi decimal e varchar; il lavoro è completato quando il dtype restituito conserva i valori 1.9 attesi senza compromettere la nuova query dei metadati.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

bug Stale
Describe the bug

Filing on behalf of some dbt Cloud customers.

Some users had some business logic that depended on the data type output of the adapter.get_columns_in_relation() method call. With 1.10, we've changed the query we ran to retrieve that information from:

describe table `dev`.`dbt_jyeo_prod_marketing`.`foo`

to

DESCRIBE EXTENDED `dev`.`dbt_jyeo_prod_marketing`.`foo` AS JSON

And I think the parsing of the results is also different.

This caused the data type output of decimal to have an extra space character (decimal(4,2) > decimal(4, 2)) and therefore break business logic where applicable.

Steps To Reproduce

Project setup:

# dbt_project.yml
name: analytics
profile: all
version: "1.0.0"

models:
  analytics:
    +materialized: table

-- macros/check.sql
{% macro check() %}
    {% set columns = adapter.get_columns_in_relation(ref('foo')) %}
    {% do print(columns) %}
    {% do print(columns[0]) %}
    {% do print(columns[0].name) %}
    {% do print(columns[0].dtype) %}
    {% do print(columns[0].dtype == 'decimal(4,2)') %}
{% endmacro %}

-- models/foo.sql
select cast(11.11 as DECIMAL(4,2)) as c

On dbt-databricks~=1.9.0

$ dbt --version && dbt build && dbt run-operation check
Core:
  - installed: 1.10.9
  - latest:    1.10.9 - Up to date!

Plugins:
  - databricks: 1.9.7 - Update available!
  - spark:      1.9.3 - Up to date!

  At least one plugin is out of date with dbt-core.
  You can find instructions for upgrading here:
  https://docs.getdbt.com/docs/installation

20:53:57  Running with dbt=1.10.9
20:53:58  Registered adapter: databricks=1.9.7
20:53:58  Found 1 model, 609 macros
20:53:58  
20:53:58  Concurrency: 1 threads (target='db')
20:53:58  
20:54:01  1 of 1 START sql table model dbt_jyeo_prod.foo ................................. [RUN]
20:54:05  1 of 1 OK created sql table model dbt_jyeo_prod.foo ............................ [OK in 4.63s]
20:54:07  
20:54:07  Finished running 1 table model in 0 hours 0 minutes and 8.47 seconds (8.47s).
20:54:07  
20:54:07  Completed successfully
20:54:07  
20:54:07  Done. PASS=1 WARN=0 ERROR=0 SKIP=0 NO-OP=0 TOTAL=1

20:54:08  Running with dbt=1.10.9
20:54:08  Registered adapter: databricks=1.9.7
20:54:08  Found 1 model, 609 macros
[<DatabricksColumn c (decimal(4,2))>]
<DatabricksColumn c (decimal(4,2))>
c
decimal(4,2)
True

On dbt-databricks~=1.10.0:

$ pip install dbt-databricks~=1.10.0
...
Installing collected packages: databricks-sql-connector, databricks-sdk, dbt-databricks
  Attempting uninstall: databricks-sql-connector
    Found existing installation: databricks-sql-connector 3.7.4
    Uninstalling databricks-sql-connector-3.7.4:
      Successfully uninstalled databricks-sql-connector-3.7.4
  Attempting uninstall: databricks-sdk
    Found existing installation: databricks-sdk 0.17.0
    Uninstalling databricks-sdk-0.17.0:
      Successfully uninstalled databricks-sdk-0.17.0
  Attempting uninstall: dbt-databricks
    Found existing installation: dbt-databricks 1.9.7
    Uninstalling dbt-databricks-1.9.7:
      Successfully uninstalled dbt-databricks-1.9.7
Successfully installed databricks-sdk-0.47.0 databricks-sql-connector-4.0.5 dbt-databricks-1.10.9

$ dbt --version && dbt build && dbt run-operation check 
Core:
  - installed: 1.10.9
  - latest:    1.10.9 - Up to date!

Plugins:
  - databricks: 1.10.9 - Up to date!
  - spark:      1.9.3  - Up to date!


20:56:08  Running with dbt=1.10.9
20:56:08  Registered adapter: databricks=1.10.9
20:56:09  Found 1 model, 686 macros
20:56:09  
20:56:09  Concurrency: 1 threads (target='db')
20:56:09  
20:56:13  1 of 1 START sql table model dbt_jyeo_prod.foo ................................. [RUN]
20:56:13  [WARNING]: Use revamped materializations based on separating create and insert.  This allows more performant column comments, as well as new column features.
You may opt into the new behavior sooner by setting `flags.use_materialization_v2` to `True` in `dbt_project.yml`.
Visit https://docs.getdbt.com/reference/global-configs/behavior-changes for more information.
20:56:16  1 of 1 OK created sql table model dbt_jyeo_prod.foo ............................ [OK in 3.39s]
20:56:16  
20:56:16  Finished running 1 table model in 0 hours 0 minutes and 7.71 seconds (7.71s).
20:56:17  
20:56:17  Completed successfully
20:56:17  
20:56:17  Done. PASS=1 WARN=0 ERROR=0 SKIP=0 NO-OP=0 TOTAL=1

20:56:17  Running with dbt=1.10.9
20:56:18  Registered adapter: databricks=1.10.9
20:56:18  Found 1 model, 686 macros
[<DatabricksColumn c (decimal(4, 2))>]
<DatabricksColumn c (decimal(4, 2))>
c
decimal(4, 2)
False
Expected behavior

I think we should stick to the 1.9 behaviour?

Screenshots and log output

If applicable, add screenshots or log output to help explain your problem.

System information

The output of dbt --version:

Core:
  - installed: 1.10.9
  - latest:    1.10.9 - Up to date!

Plugins:
  - databricks: 1.10.9 - Up to date!
  - spark:      1.9.3  - Up to date!

The operating system you're using:
macOS

The output of python --version:
3.11.9

Additional context

This also affects char types...

-- make a table that is of type varchar(10)
create or replace table dev.dbt_jyeo.varchar_test (c varchar(10));
describe table dev.dbt_jyeo.varchar_test;
Image 
describe extended dev.dbt_jyeo.varchar_test as json;
Image

One will return varchar(10) and the other returns string and will break code like:

{% if col.dtype != 'varchar(10)' %}
  raise exception...
{% endif %}
Lingua principale
Python
Stelle
391
Fork
231
Merge medio
9g 7h
PR unite (30g)
29

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di databricks/dbt-databricks

Tutte le issue di databricks/dbt-databricks

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.