[Databricks][Fusion] [BLOCKER] Existing table dropped before enforced contract validation

Aperta
  • Qualcosa non va?
#1,762 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 6 giorni

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
46/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
python, sql
Ambito
backend, databases

Direzione di ricerca

Start by tracing the Databricks table materialization and enforced-contract validation path in the adapter, then compare its operation order with dbt Core and the related issue #13654. Reproduce with an existing populated table and a contract datatype mismatch. Done means failed validation preserves the existing table, and successful replacement does not leave the target missing if creation fails.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

bug
Describe the bug

Major blocking issue: dbt Fusion drops an existing populated Databricks table before validating the model's enforced contract.

When running dbt run --select customer, Fusion executes DROP TABLE IF EXISTS against the existing target table before performing contract validation.

If validation fails, the existing table has already been dropped and is completely gone. This happens even though the model run fails and no replacement table is successfully created.

The DROP TABLE IF EXISTS also occurs on runs where the contract is valid.

This makes dbt Fusion unsafe for production use with Databricks table materializations and enforced contracts.

Steps To Reproduce
  1. Create and populate a Databricks table using a dbt model named customer, configured with materialized='table' and an enforced contract.
  2. Run the model using dbt Fusion.
  3. Observe that Fusion executes DROP TABLE IF EXISTS against the existing target table.
  4. Change the source column so that the model query returns it as string, while the enforced contract defines it as boolean.
  5. Run the model again using dbt run --select customer.
  6. Contract validation detects the datatype mismatch and the run fails.
  7. Check the target table: the original populated table has been completely dropped.

The same scenario works as expected with dbt Core: contract validation fails, but the existing table is not lost.

Expected behavior

Enforced contract validation should happen before any destructive operation against the existing target table.

If contract validation fails, the existing populated table must remain untouched and available to downstream consumers.

A successful replacement should also use a safe replacement strategy that does not leave the target table missing if validation or creation fails.

Screenshots and log output

Fusion error output:

09:58:53  [ERROR]: in model customer
09:58:53    Compilation Error in model customer
  This model has an enforced contract that failed.
  Please ensure the name, data_type, and number of columns in your contract match the columns in your model's definition.
  
  | column_name | definition_type | contract_type | mismatch_reason    |
  | ----------- | --------------- | ------------- | ------------------ |
  | source      | string          | boolean       | data type mismatch |

Observed SQL execution order:

Fusion executes:

DROP TABLE IF EXISTS ...

Followed by column/datatype checks and:

CREATE OR REPLACE TABLE ... AS SELECT ...

The contract validation fails with the datatype mismatch shown above, but the original table has already been dropped.

Fusion version:

dbt-fusion 2.0.0-preview.60

Working dbt Core comparison:

Running with dbt=1.12.6
Registered adapter: databricks=1.13.0
System information

The output of dbt --version:

dbt-fusion 2.0.0-preview.60

The operating system you're using:

Windows 11

The output of python --version:

Python 3.12.10

Additional environment details:

  • Execution: Databricks SQL Warehouse
  • Catalog: Unity Catalog
  • Adapter: Databricks
  • Adapter version: 1.13.0
  • Materialization: table
  • Enforced contract: true
  • Invocation: VS Code terminal using dbt Fusion
Additional context

This is a major blocking issue for production adoption of dbt Fusion with Databricks.

A normal enforced-contract validation failure can cause an existing production table to disappear. This can break downstream jobs, dashboards, and applications that depend on the table.

The same model and scenario work correctly with dbt Core, so this appears to be specific to the Fusion execution/materialization path.

Related issue: https://github.com/dbt-labs/dbt-core/issues/13654

That issue describes related DROP TABLE IF EXISTS behaviour on Databricks, but this report specifically highlights the destructive impact when enforced-contract validation fails.

Lingua principale
Python
Stelle
391
Fork
231
Merge medio
9g 7h
PR unite (30g)
29

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di databricks/dbt-databricks

Tutte le issue di databricks/dbt-databricks

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.