binding a schema to DAB breaks DLT pipeline for Workday

Đang mở
  • Có gì không ổn?
#3,251 5 bình luận 0 reaction 1 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

@andrewnester đang làm issue này rồi.

Từ ngày 31/7/2025.

Đánh giá

Issue này chưa được đánh giá.

Mô tả

Bug DABs No Autoclose Response Requested
Describe the issue

I'm bringing schemas under DABs management by adding schema resources to the bundle config and running databricks bundle bind... commands to import the existing schemas. This works as expected except when I try to deploy one DLT pipeline to ingest a Workday report fails. All other DLT pipelines under the impacted schemas still work fine. If I don't import or manage the particular schema that contains that pipeline I can deploy as expected.

The error is:

Error: terraform apply: exit status 1

Error: cannot update pipeline: Schema in the spec does not match the destination_schema

  with databricks_pipeline.dbjent019_01_corporate_directory,
  on bundle.tf.json line 3741, in resource.databricks_pipeline.dbjent019_01_corporate_directory:
3741:       },
Configuration

Hard to make a completely reproducible example as this depends on deploying a pipeline with Workday but I can provide snippets of the relevant sections and the setup overall.

Running databricks CLI v0.258.0, using Azure Databricks.

Example schema spec:

    staging_corporate_services:
      catalog_name: ${var.catalog}
      comment: "Managed by DABs"
      name: "${var.schema_prefix}_staging_corporate_services"
      grants:
        - principal: "ADB-Data-${var.environment_permission_name}_ENT_STAGING_SCH_READER"
          privileges: ["USE_SCHEMA"]
        - principal: "ADB-Data-ENT_READER_RESTRICTED"
          privileges: ["SELECT"]

Example pipeline that works fine:

resources:
  pipelines:
    dbjent014_01_facilities:
      permissions:
        - group_name: ${var.jobs_viewer_group}
          level: CAN_VIEW
        - group_name: ${var.jobs_run_manager_group}
          level: CAN_RUN
        - service_principal_name: ${var.jobs_control_m_sp}
          level: CAN_MANAGE
      clusters:
        - label: default
          policy_id: ${var.dlt_policy_id}
          autoscale:
            min_workers: 1
            max_workers: 5
            mode: ENHANCED
        - label: maintenance
          policy_id: ${var.dlt_policy_id}
      development: false
      continuous: false
      channel: CURRENT
      photon: false
      libraries:
        - notebook:
            path: ../jobs/delta_live_tables/corporate_services/access_card_scan_event.sql
      name: dbjent014_01_facilities
      edition: ADVANCED
      catalog: ${var.catalog}
      configuration:
        catalog: ${var.catalog}
        schema_prefix: ${var.schema_prefix}
        adbent_path: ${var.adbent_path}
      target: ${var.schema_prefix}_staging_corporate_services

Example pipeline that fails:

resources:
  pipelines:
    dbjent019_01_corporate_directory:
      permissions:
        - group_name: ${var.jobs_viewer_group}
          level: CAN_VIEW
        - group_name: ${var.jobs_run_manager_group}
          level: CAN_RUN
        - service_principal_name: ${var.jobs_control_m_sp}
          level: CAN_MANAGE
      development: false
      continuous: false
      channel: CURRENT
      photon: false

      name: dbjent019_01_corporate_directory
      edition: ADVANCED
      catalog: ${var.catalog}
      configuration:
        catalog: ${var.catalog}
        schema_prefix: ${var.schema_prefix}
        adbent_path: ${var.adbent_path}
      target: ${var.schema_prefix}_staging_corporate_services
      serverless: true
      ingestion_definition:
        connection_name: workday_prod
        objects:
          - report:
              source_url: [redacted]
              destination_catalog: ${var.catalog}
              destination_schema: ${var.schema_prefix}_staging_corporate_services
              destination_table: _workday_corporate_directory_hist
              table_configuration:
                primary_keys:
                 - employeeID
                scd_type: SCD_TYPE_2

If I change the argument on the failing pipeline from target to schema I will have to redeploy that pipeline but it will resolve the issue. I don't really want to bundle those two changes in one release though and don't see why bringing a schema under management should force that. I know target is legacy but it's all that was available when the pipeline was built.

Steps to reproduce the behavior
  1. Define a DAB resource for a schema that is the target of a Workday DLT pipeline
  2. bind the schema to your state so it's not recreated
  3. run a bundle deploy
Expected Behavior

Bundle deploy runs, maybe with a deprecation warning about schema vs target (although for all pipelines that have this, not just the Workday one)

Actual Behavior

bash```
Error: terraform apply: exit status 1

Error: cannot update pipeline: Schema in the spec does not match the destination_schema

with databricks_pipeline.dbjent019_01_corporate_directory,
on bundle.tf.json line 3741, in resource.databricks_pipeline.dbjent019_01_corporate_directory:
3741: },


### OS and CLI version
Ubuntu 22.04, cli 0.258.0

### Is this a regression?

First time trying this.

### Debug Logs

I can dump this in if required but it would take a lot of redacting and after a read through I didn't see anything relevant beyond the error already included.
Ngôn ngữ chính
Go
Star
427
Fork
267
Merge trung bình
2 ngày 1 giờ
Pull request đã merge (30 ngày)
233

Chuẩn bị môi trường

  • Có Dockerfile hoặc tệp Docker Compose
  • Có mẫu pull request
  • Không có hướng dẫn đóng góp

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của databricks/cli

Tất cả issue của databricks/cli

Issue tương tự

Thêm issue về Go

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.