[Databricks][Fusion] [BLOCKER] Existing table dropped before enforced contract validation

Open
  • Is something wrong?
#1,762 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 6 days

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
46/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
python, sql
Domain
backend, databases

Research direction

Start by tracing the Databricks table materialization and enforced-contract validation path in the adapter, then compare its operation order with dbt Core and the related issue #13654. Reproduce with an existing populated table and a contract datatype mismatch. Done means failed validation preserves the existing table, and successful replacement does not leave the target missing if creation fails.

Written by the indexing model from the issue text.

Description

bug
Describe the bug

Major blocking issue: dbt Fusion drops an existing populated Databricks table before validating the model's enforced contract.

When running dbt run --select customer, Fusion executes DROP TABLE IF EXISTS against the existing target table before performing contract validation.

If validation fails, the existing table has already been dropped and is completely gone. This happens even though the model run fails and no replacement table is successfully created.

The DROP TABLE IF EXISTS also occurs on runs where the contract is valid.

This makes dbt Fusion unsafe for production use with Databricks table materializations and enforced contracts.

Steps To Reproduce
  1. Create and populate a Databricks table using a dbt model named customer, configured with materialized='table' and an enforced contract.
  2. Run the model using dbt Fusion.
  3. Observe that Fusion executes DROP TABLE IF EXISTS against the existing target table.
  4. Change the source column so that the model query returns it as string, while the enforced contract defines it as boolean.
  5. Run the model again using dbt run --select customer.
  6. Contract validation detects the datatype mismatch and the run fails.
  7. Check the target table: the original populated table has been completely dropped.

The same scenario works as expected with dbt Core: contract validation fails, but the existing table is not lost.

Expected behavior

Enforced contract validation should happen before any destructive operation against the existing target table.

If contract validation fails, the existing populated table must remain untouched and available to downstream consumers.

A successful replacement should also use a safe replacement strategy that does not leave the target table missing if validation or creation fails.

Screenshots and log output

Fusion error output:

09:58:53  [ERROR]: in model customer
09:58:53    Compilation Error in model customer
  This model has an enforced contract that failed.
  Please ensure the name, data_type, and number of columns in your contract match the columns in your model's definition.
  
  | column_name | definition_type | contract_type | mismatch_reason    |
  | ----------- | --------------- | ------------- | ------------------ |
  | source      | string          | boolean       | data type mismatch |

Observed SQL execution order:

Fusion executes:

DROP TABLE IF EXISTS ...

Followed by column/datatype checks and:

CREATE OR REPLACE TABLE ... AS SELECT ...

The contract validation fails with the datatype mismatch shown above, but the original table has already been dropped.

Fusion version:

dbt-fusion 2.0.0-preview.60

Working dbt Core comparison:

Running with dbt=1.12.6
Registered adapter: databricks=1.13.0
System information

The output of dbt --version:

dbt-fusion 2.0.0-preview.60

The operating system you're using:

Windows 11

The output of python --version:

Python 3.12.10

Additional environment details:

  • Execution: Databricks SQL Warehouse
  • Catalog: Unity Catalog
  • Adapter: Databricks
  • Adapter version: 1.13.0
  • Materialization: table
  • Enforced contract: true
  • Invocation: VS Code terminal using dbt Fusion
Additional context

This is a major blocking issue for production adoption of dbt Fusion with Databricks.

A normal enforced-contract validation failure can cause an existing production table to disappear. This can break downstream jobs, dashboards, and applications that depend on the table.

The same model and scenario work correctly with dbt Core, so this appears to be specific to the Fusion execution/materialization path.

Related issue: https://github.com/dbt-labs/dbt-core/issues/13654

That issue describes related DROP TABLE IF EXISTS behaviour on Databricks, but this report specifically highlights the destructive impact when enforced-contract validation fails.

Dominant language
Python
Stars
391
Forks
231
Avg merge
9d 7h
Merged PRs (30d)
29

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from databricks/dbt-databricks

All issues in databricks/dbt-databricks

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.