[Bug]: CLI/Bundles strict validation rejects officially documented 'meta_ads_options' in Lakeflow Connect pipelines (Similar to #4883)

Open
  • Is something wrong?
#5,253 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
68/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Quiet
Tech stack
go
Domain
cli

Research direction

Start by reproducing the bundle deploy or pipelines update with the databricks.yml/resources configuration shown, then trace the CLI schema validation for connector_options and meta_ads_options. Done means the documented block and its nested parameters are accepted, preserved in metadata.json, and passed to the backend without validation errors or stripping.

Written by the indexing model from the issue text.

Description

DABs Stale
Describe the issue

Description:
Similar to the strict OpenAPI schema validation issues reported in #4883, the Databricks CLI currently fails to compile and deploy valid, officially documented configurations for the Lakeflow Connect Meta Ads connector.

Specifically, the CLI rejects the meta_ads_options block nested under connector_options during a bundle deploy or pipelines update, effectively locking users out of pulling conversion data (action breakdowns) via IaC.

Official Documentation Reference:
According to the official Lakeflow Connect documentation for Meta Ads, users must configure action_breakdowns and action_attribution_windows inside the meta_ads_options block to retrieve conversion data:
Ingest ad_insights with meta_ads_options

Technical Explanation & Troubleshooting Journey:

When attempting to deploy a Delta Live Tables / Lakeflow Connect pipeline via Databricks Asset Bundles, we defined the ad_insights table configuration to include the meta_ads_options block exactly as documented.

When executing databricks bundle deploy, the local CLI schema validator throws an unknown field: meta_ads_options error/warning and strips the configuration from the payload before sending it to the backend. Attempting to bypass the bundle using databricks pipelines update --json @file.json results in the exact same schema validation stripping the block.

To prove this was strictly a CLI validation bug and not a backend API limitation, we bypassed the CLI's validation entirely by using the raw REST API passthrough: databricks api put /api/2.0/pipelines/ --json @update.json.

This workaround successfully hit the backend REST API, and the Databricks UI instantly reflected the injected action_breakdowns and action_attribution_windows configurations, proving the control plane actively accepts this schema.

However, this introduces an unpassable "Catch-22" for teams relying on CI/CD and Infrastructure as Code:
Because the pipeline is managed by a Databricks Asset Bundle, the overarching state is dictated by the bundle's metadata.json state file. Since the CLI compiler stripped the meta_ads_options during the initial deployment, the official bundle state does not contain the configuration. When we manually trigger the pipeline in the UI (after successfully injecting the JSON via the API), the DLT engine compares the live configuration against the metadata.json state file, detects a drift, and immediately auto-reverts the pipeline to the stripped configuration.

Because the CLI rejects the valid code, it cannot be added to the state file. Because it is not in the state file, the backend auto-erases any manual API overrides. We are completely blocked from configuring Meta Ads conversion ingestion via the CLI.

Steps to reproduce the behavior

Attempt to run databricks bundle deploy with the following valid configuration in the databricks.yml or resources file:

resources:
  pipelines:
    meta_ads_ingestion_pipeline:
      name: meta_ads_ingestion_pipeline
      ingestion_definition:
        connection_name: meta_ads_connection
        objects:
          - table:
              source_schema: act_123456789
              source_table: ad_insights
              destination_catalog: raw_catalog
              destination_schema: meta_ads
              destination_table: ad_insights_action_breakdowns
              table_configuration:
                scd_type: SCD_TYPE_1
              connector_options:
                meta_ads_options:
                  level: ad
                  action_breakdowns:
                    - action_type
                  action_attribution_windows:
                    - 7d_click
                    - 1d_view
Expected Behavior

The Databricks CLI should recognize meta_ads_options (and its nested parameters) as a valid block under connector_options for Meta Ads ingestion pipelines, compile it into the bundle state, and successfully pass it to the backend without throwing validation errors or stripping the payload.

Actual Behavior

When executing databricks bundle deploy, the local CLI schema validator throws an unknown field: meta_ads_options error/warning and strips the configuration from the payload before sending it to the backend.

Environment and CLI version

Databricks CLI Version: v0.299.1
Target Environment: Azure Databricks (Lakeflow Connect Preview)

Is this a regression?

N/A

Dominant language
Go
Stars
427
Forks
267
Avg merge
2d 1h
Merged PRs (30d)
233

Getting set up

  • Ships a Dockerfile or Docker Compose file
  • Has a pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from databricks/cli

All issues in databricks/cli

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.