binding a schema to DAB breaks DLT pipeline for Workday

Abierto
  • ¿Algo no está bien?
#3,251 5 comentarios 0 reacciones 1 asignado Ver en GitHub

Los mantenedores suelen responder en 1 día

@andrewnester ya está trabajando en esto.

Desde el 31/7/2025.

Evaluación

Este issue todavía no se ha evaluado.

Descripción

Bug DABs No Autoclose Response Requested
Describe the issue

I'm bringing schemas under DABs management by adding schema resources to the bundle config and running databricks bundle bind... commands to import the existing schemas. This works as expected except when I try to deploy one DLT pipeline to ingest a Workday report fails. All other DLT pipelines under the impacted schemas still work fine. If I don't import or manage the particular schema that contains that pipeline I can deploy as expected.

The error is:

Error: terraform apply: exit status 1

Error: cannot update pipeline: Schema in the spec does not match the destination_schema

  with databricks_pipeline.dbjent019_01_corporate_directory,
  on bundle.tf.json line 3741, in resource.databricks_pipeline.dbjent019_01_corporate_directory:
3741:       },
Configuration

Hard to make a completely reproducible example as this depends on deploying a pipeline with Workday but I can provide snippets of the relevant sections and the setup overall.

Running databricks CLI v0.258.0, using Azure Databricks.

Example schema spec:

    staging_corporate_services:
      catalog_name: ${var.catalog}
      comment: "Managed by DABs"
      name: "${var.schema_prefix}_staging_corporate_services"
      grants:
        - principal: "ADB-Data-${var.environment_permission_name}_ENT_STAGING_SCH_READER"
          privileges: ["USE_SCHEMA"]
        - principal: "ADB-Data-ENT_READER_RESTRICTED"
          privileges: ["SELECT"]

Example pipeline that works fine:

resources:
  pipelines:
    dbjent014_01_facilities:
      permissions:
        - group_name: ${var.jobs_viewer_group}
          level: CAN_VIEW
        - group_name: ${var.jobs_run_manager_group}
          level: CAN_RUN
        - service_principal_name: ${var.jobs_control_m_sp}
          level: CAN_MANAGE
      clusters:
        - label: default
          policy_id: ${var.dlt_policy_id}
          autoscale:
            min_workers: 1
            max_workers: 5
            mode: ENHANCED
        - label: maintenance
          policy_id: ${var.dlt_policy_id}
      development: false
      continuous: false
      channel: CURRENT
      photon: false
      libraries:
        - notebook:
            path: ../jobs/delta_live_tables/corporate_services/access_card_scan_event.sql
      name: dbjent014_01_facilities
      edition: ADVANCED
      catalog: ${var.catalog}
      configuration:
        catalog: ${var.catalog}
        schema_prefix: ${var.schema_prefix}
        adbent_path: ${var.adbent_path}
      target: ${var.schema_prefix}_staging_corporate_services

Example pipeline that fails:

resources:
  pipelines:
    dbjent019_01_corporate_directory:
      permissions:
        - group_name: ${var.jobs_viewer_group}
          level: CAN_VIEW
        - group_name: ${var.jobs_run_manager_group}
          level: CAN_RUN
        - service_principal_name: ${var.jobs_control_m_sp}
          level: CAN_MANAGE
      development: false
      continuous: false
      channel: CURRENT
      photon: false

      name: dbjent019_01_corporate_directory
      edition: ADVANCED
      catalog: ${var.catalog}
      configuration:
        catalog: ${var.catalog}
        schema_prefix: ${var.schema_prefix}
        adbent_path: ${var.adbent_path}
      target: ${var.schema_prefix}_staging_corporate_services
      serverless: true
      ingestion_definition:
        connection_name: workday_prod
        objects:
          - report:
              source_url: [redacted]
              destination_catalog: ${var.catalog}
              destination_schema: ${var.schema_prefix}_staging_corporate_services
              destination_table: _workday_corporate_directory_hist
              table_configuration:
                primary_keys:
                 - employeeID
                scd_type: SCD_TYPE_2

If I change the argument on the failing pipeline from target to schema I will have to redeploy that pipeline but it will resolve the issue. I don't really want to bundle those two changes in one release though and don't see why bringing a schema under management should force that. I know target is legacy but it's all that was available when the pipeline was built.

Steps to reproduce the behavior
  1. Define a DAB resource for a schema that is the target of a Workday DLT pipeline
  2. bind the schema to your state so it's not recreated
  3. run a bundle deploy
Expected Behavior

Bundle deploy runs, maybe with a deprecation warning about schema vs target (although for all pipelines that have this, not just the Workday one)

Actual Behavior

bash```
Error: terraform apply: exit status 1

Error: cannot update pipeline: Schema in the spec does not match the destination_schema

with databricks_pipeline.dbjent019_01_corporate_directory,
on bundle.tf.json line 3741, in resource.databricks_pipeline.dbjent019_01_corporate_directory:
3741: },


### OS and CLI version
Ubuntu 22.04, cli 0.258.0

### Is this a regression?

First time trying this.

### Debug Logs

I can dump this in if required but it would take a lot of redacting and after a read through I didn't see anything relevant beyond the error already included.
Lenguaje dominante
Go
Estrellas
427
Forks
267
Merge medio
2 d 1 h
PR fusionados (30 d)
233

Preparar el entorno

  • Incluye un Dockerfile o un archivo de Docker Compose
  • Tiene una plantilla de pull request
  • Sin guía de contribución

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de databricks/cli

Todos los issues de databricks/cli

Issues similares

Más issues de Go

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.