PowerBI Task DAB deployment fails if table does not exist

Open
  • Is something wrong?
#4,370 4 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
48/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Quiet
Tech stack
go
Domain
cli, devops

Research direction

Start with the asset bundle definition and run databricks bundle deploy using a referenced table that does not yet exist. Inspect the generated bundle.tf.json resource and the Terraform apply error to trace the PowerBI task validation. Done means deployment succeeds before the table exists, while the later pipeline run can create it.

Written by the indexing model from the issue text.

Description

Bug DABs
Describe the issue

When deploying a PowerBI Task the asset bundles validates if the table we plan to push to PowerBI exists. While this check makes sense for incremental workflows for the initial deployment of the job this is not useful since the table first has to be created by the job which is firstly deployed to production.

Configuration

Create a PowerBI task via DAB and reference a table that does not yet exists. If I want to deploy this asset bundle it will fail because the table for the PowerBI Task does not yet exits.

Steps to reproduce the behavior
  1. Create a asset bundle definition with a PowerBI Task
  2. Reference a Table that does not yet exists.
  3. Run databricks bundle deploy
Expected Behavior

Pipeline will be deployed even those table does not yet exists. Since it will be created during the pipline run. Since we have a dbt task running before the PowerBI task.

Actual Behavior

Deployment fails because table does not exist during deployment time but will exist during run time of the pipeline.

OS and CLI version

Windwos
Databricks CLI v0.272.0

Is this a regression?

I don't know tried this the first time.

Debug Logs

Deploying resources...
11:06:06 Debug: Apply pid=12544 mutator=terraform.Apply
Error: terraform apply: exit status 1

Error: cannot update job: 'Run as' user is missing permissions: USE CATALOG on catalog tindp_bi_dev, USE SCHEMA on schema dev_mart or SELECT on table some_insights__obt_viral_alerts

with databricks_job.some_insights__transform_hourly,
on bundle.tf.json line 183, in resource.databricks_job.some_insights__transform_hourly:
183: },

Dominant language
Go
Stars
427
Forks
267
Avg merge
2d 1h
Merged PRs (30d)
233

Getting set up

  • Ships a Dockerfile or Docker Compose file
  • Has a pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from databricks/cli

All issues in databricks/cli

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.