Job with Table Trigger not allowed for tables that do not exist in Unity Catalog

Open
  • Is something wrong?
#4,437 5 comments 11 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
52/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
go
Domain
cli, devops

Research direction

Reproduce the failure with the shown bundle configuration using databricks bundle deploy and inspect the Terraform apply error. Trace how the table trigger is validated during deployment, then verify that a missing Unity Catalog table produces a warning while the job is still deployed successfully.

Written by the indexing model from the issue text.

Description

DABs Enhancement
Describe the issue

In the current implementation, I am unable to deploy a job with a table trigger that depends on one (or more) tables that are not yet available in Unity Catalog.
This hinders CI/CD scenarios in which the table is not yet created in the upper environment as the table creation is relying on a run that is yet to happen

Configuration

A DAB with the below configuration is not deployable unless the table demo.table.trigger already exists

  jobs:
    demo_job:
      name: "Demo - ${bundle.target}"
      
      tasks:
        - task_key: demo
          notebook_task:
            notebook_path: ../notebooks/run_notebook.py
      
      trigger:
        pause_status: UNPAUSED
        table_update:
          table_names:
            - demo.table.trigger
Steps to reproduce the behavior
  1. Run databricks bundle deploy ...
  2. See error
Expected Behavior

I would expect a WARNING instead of an error.
It makes sense to warn the user that their trigger might not ever happen - but it does not make sense to not allow it to be deployed.

Actual Behavior

Below error message, stating that I can not create a trigger on a job to a table that does not exist yet

Error: terraform apply: exit status 1
Error: cannot create job: The table 'demo.table.trigger' does not exist. Please create a table in Unity Catalog and grant SELECT privilege to [email protected]
OS and CLI version

Databricks CLI v0.283.0 on Windows

Is this a regression?

Did this work in a previous version of the CLI? If so, which versions did you try?

Debug Logs

Output logs if you run the command with debug logs enabled. Example: databricks bundle deploy --log-level=debug. Redact if needed

Dominant language
Go
Stars
427
Forks
267
Avg merge
2d 1h
Merged PRs (30d)
233

Getting set up

  • Ships a Dockerfile or Docker Compose file
  • Has a pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from databricks/cli

All issues in databricks/cli

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.