binding a schema to DAB breaks DLT pipeline for Workday

Aberta
  • Algo errado?
#3,251 5 comentários 0 reações 1 responsável Ver no GitHub

Mantenedores costumam responder em até 1 dia

@andrewnester já está trabalhando nisso.

Desde 31/7/2025.

Avaliação

Esta issue ainda não foi avaliada.

Descrição

Bug DABs No Autoclose Response Requested
Describe the issue

I'm bringing schemas under DABs management by adding schema resources to the bundle config and running databricks bundle bind... commands to import the existing schemas. This works as expected except when I try to deploy one DLT pipeline to ingest a Workday report fails. All other DLT pipelines under the impacted schemas still work fine. If I don't import or manage the particular schema that contains that pipeline I can deploy as expected.

The error is:

Error: terraform apply: exit status 1

Error: cannot update pipeline: Schema in the spec does not match the destination_schema

  with databricks_pipeline.dbjent019_01_corporate_directory,
  on bundle.tf.json line 3741, in resource.databricks_pipeline.dbjent019_01_corporate_directory:
3741:       },
Configuration

Hard to make a completely reproducible example as this depends on deploying a pipeline with Workday but I can provide snippets of the relevant sections and the setup overall.

Running databricks CLI v0.258.0, using Azure Databricks.

Example schema spec:

    staging_corporate_services:
      catalog_name: ${var.catalog}
      comment: "Managed by DABs"
      name: "${var.schema_prefix}_staging_corporate_services"
      grants:
        - principal: "ADB-Data-${var.environment_permission_name}_ENT_STAGING_SCH_READER"
          privileges: ["USE_SCHEMA"]
        - principal: "ADB-Data-ENT_READER_RESTRICTED"
          privileges: ["SELECT"]

Example pipeline that works fine:

resources:
  pipelines:
    dbjent014_01_facilities:
      permissions:
        - group_name: ${var.jobs_viewer_group}
          level: CAN_VIEW
        - group_name: ${var.jobs_run_manager_group}
          level: CAN_RUN
        - service_principal_name: ${var.jobs_control_m_sp}
          level: CAN_MANAGE
      clusters:
        - label: default
          policy_id: ${var.dlt_policy_id}
          autoscale:
            min_workers: 1
            max_workers: 5
            mode: ENHANCED
        - label: maintenance
          policy_id: ${var.dlt_policy_id}
      development: false
      continuous: false
      channel: CURRENT
      photon: false
      libraries:
        - notebook:
            path: ../jobs/delta_live_tables/corporate_services/access_card_scan_event.sql
      name: dbjent014_01_facilities
      edition: ADVANCED
      catalog: ${var.catalog}
      configuration:
        catalog: ${var.catalog}
        schema_prefix: ${var.schema_prefix}
        adbent_path: ${var.adbent_path}
      target: ${var.schema_prefix}_staging_corporate_services

Example pipeline that fails:

resources:
  pipelines:
    dbjent019_01_corporate_directory:
      permissions:
        - group_name: ${var.jobs_viewer_group}
          level: CAN_VIEW
        - group_name: ${var.jobs_run_manager_group}
          level: CAN_RUN
        - service_principal_name: ${var.jobs_control_m_sp}
          level: CAN_MANAGE
      development: false
      continuous: false
      channel: CURRENT
      photon: false

      name: dbjent019_01_corporate_directory
      edition: ADVANCED
      catalog: ${var.catalog}
      configuration:
        catalog: ${var.catalog}
        schema_prefix: ${var.schema_prefix}
        adbent_path: ${var.adbent_path}
      target: ${var.schema_prefix}_staging_corporate_services
      serverless: true
      ingestion_definition:
        connection_name: workday_prod
        objects:
          - report:
              source_url: [redacted]
              destination_catalog: ${var.catalog}
              destination_schema: ${var.schema_prefix}_staging_corporate_services
              destination_table: _workday_corporate_directory_hist
              table_configuration:
                primary_keys:
                 - employeeID
                scd_type: SCD_TYPE_2

If I change the argument on the failing pipeline from target to schema I will have to redeploy that pipeline but it will resolve the issue. I don't really want to bundle those two changes in one release though and don't see why bringing a schema under management should force that. I know target is legacy but it's all that was available when the pipeline was built.

Steps to reproduce the behavior
  1. Define a DAB resource for a schema that is the target of a Workday DLT pipeline
  2. bind the schema to your state so it's not recreated
  3. run a bundle deploy
Expected Behavior

Bundle deploy runs, maybe with a deprecation warning about schema vs target (although for all pipelines that have this, not just the Workday one)

Actual Behavior

bash```
Error: terraform apply: exit status 1

Error: cannot update pipeline: Schema in the spec does not match the destination_schema

with databricks_pipeline.dbjent019_01_corporate_directory,
on bundle.tf.json line 3741, in resource.databricks_pipeline.dbjent019_01_corporate_directory:
3741: },


### OS and CLI version
Ubuntu 22.04, cli 0.258.0

### Is this a regression?

First time trying this.

### Debug Logs

I can dump this in if required but it would take a lot of redacting and after a read through I didn't see anything relevant beyond the error already included.
Linguagem predominante
Go
Estrelas
427
Forks
267
Merge médio
2d 1h
PRs com merge (30d)
233

Preparar o ambiente

  • Inclui um Dockerfile ou arquivo Docker Compose
  • Tem um modelo de pull request
  • Sem guia de contribuição

Primeiros passos

  1. Leia a issue inteira e depois o guia de contribuição do projeto.
  2. Comente na issue dizendo que vai assumir — evita que duas pessoas façam o mesmo trabalho.
  3. Faça um fork do repositório e trabalhe em uma branch.
  4. Abra um pull request que referencie o número da issue.

Mais de databricks/cli

Todas as issues de databricks/cli

Issues semelhantes

Mais issues de Go

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.