DAB deploys with zero actual config changes still trigger a SETTINGS_CHANGE cancel+restart on continuous pipelines

Open
#6,745 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
42/100
Issue type
Bug
Clarity
Needs clarification
Activity status
Active
Tech stack
go
Domain
cli

Research direction

Start by reproducing with the continuous pipeline configuration and capture a redacted bundle plan -o json -t <target> plus the pipeline event log or update history. Compare the resolved bundle configuration with the remote pipeline state to isolate the unregistered input-only or computed field. Done means repeated deploys with no configuration changes no longer cancel and restart the continuous pipeline.

Written by the indexing model from the issue text.

Description

Describe the issue

A bundle deploy that makes no actual configuration change to a continuous
pipeline still causes the Pipelines service to cancel the pipeline's active
update and start a new one, with reason SETTINGS_CHANGE. This happens on
effectively every deploy of a bundle containing a continuous pipeline, even
when that specific pipeline's resource definition is byte-for-byte unchanged
between deploys.

This looks like the same class of bug as #6076, #6060 and #6315 — an
input-only or computed field on the pipeline resource that isn't registered
as such, so the direct engine perpetually diffs bundle state against a
remote value that structurally can never match, and replays an update every
run. Unlike those, the field responsible hasn't been identified yet; filing
this to track it and get guidance on capturing the right evidence.

Configuration

Minimal repro shape (the exact field causing the perpetual diff is not yet
isolated — happens on any continuous pipeline with a libraries.notebook
based DLT pipeline, no gateway/ingestion-specific config required):

resources:
  pipelines:
    example_stream:
      name: example_stream
      catalog: my_catalog
      target: my_schema
      continuous: true
      development: false
      channel: CURRENT
      edition: PRO
      libraries:
        - notebook:
            path: ../src/pipelines/example/pipeline
Steps to reproduce the behavior
  1. databricks bundle deploy -t <target> with a continuous pipeline in the bundle, no source changes since the last deploy.
  2. Observe the pipeline's event log / update history immediately after deploy.
  3. See a new update start with cause: SETTINGS_CHANGE, cancelling whatever update was previously running.
Expected Behavior

No update call (and therefore no cancel/restart) should occur against a
continuous pipeline when nothing about its resolved configuration actually
changed.

Actual Behavior

The pipeline is cancelled and a new update starts on every deploy.

OS and CLI version

CLI v1.17.0, Linux.

Is this a regression?

Yes, in scope: one continuous ingestion pipeline (Lakeflow CDC gateway) already
showed this behavior under the Terraform-based engine. After migrating this
bundle to the direct deployment engine, every continuous pipeline in the
bundle now exhibits the same behavior — not just that one.

Detailed plan

Can provide a redacted bundle plan -o json -t <target> on request — happy
to capture it if a maintainer confirms this is the most useful next
diagnostic, and to redact accordingly.

Debug Logs

Can provide on request, redacted.

Dominant language
Go
Stars
396
Forks
233
Avg merge
2d 52m
Merged PRs (30d)
276

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from databricks/cli

All issues in databricks/cli

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.