[Bug]: CLI/Bundles strict validation rejects officially documented 'meta_ads_options' in Lakeflow Connect pipelines (Similar to #4883)

首先，使用所示的 databricks.yml/resources 設定重現 bundle deploy 或 pipelines update，然後追蹤 CLI 對 connector_options 和 meta_ads_options 的 schema 驗證。完成的標準是，文件中的區塊及其巢狀參數能夠被接受、保留在 metadata.json 中，並且在沒有驗證錯誤或遭到剝除的情況下傳遞給後端。

Describe the issue

Description:
Similar to the strict OpenAPI schema validation issues reported in #4883, the Databricks CLI currently fails to compile and deploy valid, officially documented configurations for the Lakeflow Connect Meta Ads connector.

Specifically, the CLI rejects the meta_ads_options block nested under connector_options during a bundle deploy or pipelines update, effectively locking users out of pulling conversion data (action breakdowns) via IaC.

Official Documentation Reference:
According to the official Lakeflow Connect documentation for Meta Ads, users must configure action_breakdowns and action_attribution_windows inside the meta_ads_options block to retrieve conversion data:
Ingest ad_insights with meta_ads_options

Technical Explanation & Troubleshooting Journey:

When attempting to deploy a Delta Live Tables / Lakeflow Connect pipeline via Databricks Asset Bundles, we defined the ad_insights table configuration to include the meta_ads_options block exactly as documented.

When executing databricks bundle deploy, the local CLI schema validator throws an unknown field: meta_ads_options error/warning and strips the configuration from the payload before sending it to the backend. Attempting to bypass the bundle using databricks pipelines update --json @file.json results in the exact same schema validation stripping the block.

To prove this was strictly a CLI validation bug and not a backend API limitation, we bypassed the CLI's validation entirely by using the raw REST API passthrough: databricks api put /api/2.0/pipelines/ --json @update.json.

This workaround successfully hit the backend REST API, and the Databricks UI instantly reflected the injected action_breakdowns and action_attribution_windows configurations, proving the control plane actively accepts this schema.

However, this introduces an unpassable "Catch-22" for teams relying on CI/CD and Infrastructure as Code:
Because the pipeline is managed by a Databricks Asset Bundle, the overarching state is dictated by the bundle's metadata.json state file. Since the CLI compiler stripped the meta_ads_options during the initial deployment, the official bundle state does not contain the configuration. When we manually trigger the pipeline in the UI (after successfully injecting the JSON via the API), the DLT engine compares the live configuration against the metadata.json state file, detects a drift, and immediately auto-reverts the pipeline to the stripped configuration.

Because the CLI rejects the valid code, it cannot be added to the state file. Because it is not in the state file, the backend auto-erases any manual API overrides. We are completely blocked from configuring Meta Ads conversion ingestion via the CLI.

Steps to reproduce the behavior

Attempt to run databricks bundle deploy with the following valid configuration in the databricks.yml or resources file:

resources:
  pipelines:
    meta_ads_ingestion_pipeline:
      name: meta_ads_ingestion_pipeline
      ingestion_definition:
        connection_name: meta_ads_connection
        objects:
          - table:
              source_schema: act_123456789
              source_table: ad_insights
              destination_catalog: raw_catalog
              destination_schema: meta_ads
              destination_table: ad_insights_action_breakdowns
              table_configuration:
                scd_type: SCD_TYPE_1
              connector_options:
                meta_ads_options:
                  level: ad
                  action_breakdowns:
                    - action_type
                  action_attribution_windows:
                    - 7d_click
                    - 1d_view

Expected Behavior

The Databricks CLI should recognize meta_ads_options (and its nested parameters) as a valid block under connector_options for Meta Ads ingestion pipelines, compile it into the bundle state, and successfully pass it to the backend without throwing validation errors or stripping the payload.

Actual Behavior

When executing databricks bundle deploy, the local CLI schema validator throws an unknown field: meta_ads_options error/warning and strips the configuration from the payload before sending it to the backend.

Environment and CLI version

Databricks CLI Version: v0.299.1
Target Environment: Azure Databricks (Lakeflow Connect Preview)

Is this a regression?

N/A