[Bug]: CLI/Bundles strict validation rejects officially documented 'meta_ads_options' in Lakeflow Connect pipelines (Similar to #4883)

Offen
  • Stimmt etwas nicht?
#5,253 1 Kommentar 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen

Maintainer antworten meist innerhalb von 1 Tag

Dieses Issue hat noch niemand übernommen.

Bewertung

Schwierigkeit
3/5
Geschätzter Aufwand
1-2 Tage
Anfängerfreundlichkeit
68/100
Issue-Typ
Bug
Klarheit
Größtenteils klar
Aktivitätsstatus
Ruhig
Tech-Stack
go
Bereich
cli

Rechercherichtung

Beginne damit, das Bundle-Deployment oder das Pipeline-Update mit der gezeigten databricks.yml/resources-Konfiguration zu reproduzieren, und verfolge dann die CLI-Schemavalidierung für connector_options und meta_ads_options. Das Ziel ist erreicht, wenn der dokumentierte Block und seine verschachtelten Parameter akzeptiert, in metadata.json erhalten und ohne Validierungsfehler oder Entfernung an das Backend übergeben werden.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Beschreibung

DABs Stale
Describe the issue

Description:
Similar to the strict OpenAPI schema validation issues reported in #4883, the Databricks CLI currently fails to compile and deploy valid, officially documented configurations for the Lakeflow Connect Meta Ads connector.

Specifically, the CLI rejects the meta_ads_options block nested under connector_options during a bundle deploy or pipelines update, effectively locking users out of pulling conversion data (action breakdowns) via IaC.

Official Documentation Reference:
According to the official Lakeflow Connect documentation for Meta Ads, users must configure action_breakdowns and action_attribution_windows inside the meta_ads_options block to retrieve conversion data:
Ingest ad_insights with meta_ads_options

Technical Explanation & Troubleshooting Journey:

When attempting to deploy a Delta Live Tables / Lakeflow Connect pipeline via Databricks Asset Bundles, we defined the ad_insights table configuration to include the meta_ads_options block exactly as documented.

When executing databricks bundle deploy, the local CLI schema validator throws an unknown field: meta_ads_options error/warning and strips the configuration from the payload before sending it to the backend. Attempting to bypass the bundle using databricks pipelines update --json @file.json results in the exact same schema validation stripping the block.

To prove this was strictly a CLI validation bug and not a backend API limitation, we bypassed the CLI's validation entirely by using the raw REST API passthrough: databricks api put /api/2.0/pipelines/ --json @update.json.

This workaround successfully hit the backend REST API, and the Databricks UI instantly reflected the injected action_breakdowns and action_attribution_windows configurations, proving the control plane actively accepts this schema.

However, this introduces an unpassable "Catch-22" for teams relying on CI/CD and Infrastructure as Code:
Because the pipeline is managed by a Databricks Asset Bundle, the overarching state is dictated by the bundle's metadata.json state file. Since the CLI compiler stripped the meta_ads_options during the initial deployment, the official bundle state does not contain the configuration. When we manually trigger the pipeline in the UI (after successfully injecting the JSON via the API), the DLT engine compares the live configuration against the metadata.json state file, detects a drift, and immediately auto-reverts the pipeline to the stripped configuration.

Because the CLI rejects the valid code, it cannot be added to the state file. Because it is not in the state file, the backend auto-erases any manual API overrides. We are completely blocked from configuring Meta Ads conversion ingestion via the CLI.

Steps to reproduce the behavior

Attempt to run databricks bundle deploy with the following valid configuration in the databricks.yml or resources file:

resources:
  pipelines:
    meta_ads_ingestion_pipeline:
      name: meta_ads_ingestion_pipeline
      ingestion_definition:
        connection_name: meta_ads_connection
        objects:
          - table:
              source_schema: act_123456789
              source_table: ad_insights
              destination_catalog: raw_catalog
              destination_schema: meta_ads
              destination_table: ad_insights_action_breakdowns
              table_configuration:
                scd_type: SCD_TYPE_1
              connector_options:
                meta_ads_options:
                  level: ad
                  action_breakdowns:
                    - action_type
                  action_attribution_windows:
                    - 7d_click
                    - 1d_view
Expected Behavior

The Databricks CLI should recognize meta_ads_options (and its nested parameters) as a valid block under connector_options for Meta Ads ingestion pipelines, compile it into the bundle state, and successfully pass it to the backend without throwing validation errors or stripping the payload.

Actual Behavior

When executing databricks bundle deploy, the local CLI schema validator throws an unknown field: meta_ads_options error/warning and strips the configuration from the payload before sending it to the backend.

Environment and CLI version

Databricks CLI Version: v0.299.1
Target Environment: Azure Databricks (Lakeflow Connect Preview)

Is this a regression?

N/A

Vorherrschende Sprache
Go
Sterne
427
Forks
267
Ø Merge
2 T. 1 Std.
Gemergte PRs (30 T.)
233

Entwicklungsumgebung

  • Enthält ein Dockerfile oder eine Docker-Compose-Datei
  • Hat eine Pull-Request-Vorlage
  • Kein Beitragsleitfaden

Erste Schritte

  1. Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
  2. Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
  3. Forken Sie das Repository und arbeiten Sie in einem Branch.
  4. Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.

Mehr aus databricks/cli

Alle Issues in databricks/cli

Ähnliche Issues

Weitere Issues zu Go

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.