DAB deploys with zero actual config changes still trigger a SETTINGS_CHANGE cancel+restart on continuous pipelines
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 42/100
Research direction
Start by reproducing with the continuous pipeline configuration and capture a redacted bundle plan -o json -t <target> plus the pipeline event log or update history. Compare the resolved bundle configuration with the remote pipeline state to isolate the unregistered input-only or computed field. Done means repeated deploys with no configuration changes no longer cancel and restart the continuous pipeline.
Written by the indexing model from the issue text.
Description
Describe the issue
A bundle deploy that makes no actual configuration change to a continuous
pipeline still causes the Pipelines service to cancel the pipeline's active
update and start a new one, with reason SETTINGS_CHANGE. This happens on
effectively every deploy of a bundle containing a continuous pipeline, even
when that specific pipeline's resource definition is byte-for-byte unchanged
between deploys.
This looks like the same class of bug as #6076, #6060 and #6315 — an
input-only or computed field on the pipeline resource that isn't registered
as such, so the direct engine perpetually diffs bundle state against a
remote value that structurally can never match, and replays an update every
run. Unlike those, the field responsible hasn't been identified yet; filing
this to track it and get guidance on capturing the right evidence.
Configuration
Minimal repro shape (the exact field causing the perpetual diff is not yet
isolated — happens on any continuous pipeline with a libraries.notebook
based DLT pipeline, no gateway/ingestion-specific config required):
resources:
pipelines:
example_stream:
name: example_stream
catalog: my_catalog
target: my_schema
continuous: true
development: false
channel: CURRENT
edition: PRO
libraries:
- notebook:
path: ../src/pipelines/example/pipeline
Steps to reproduce the behavior
databricks bundle deploy -t <target>with a continuous pipeline in the bundle, no source changes since the last deploy.- Observe the pipeline's event log / update history immediately after deploy.
- See a new update start with
cause: SETTINGS_CHANGE, cancelling whatever update was previously running.
Expected Behavior
No update call (and therefore no cancel/restart) should occur against a
continuous pipeline when nothing about its resolved configuration actually
changed.
Actual Behavior
The pipeline is cancelled and a new update starts on every deploy.
OS and CLI version
CLI v1.17.0, Linux.
Is this a regression?
Yes, in scope: one continuous ingestion pipeline (Lakeflow CDC gateway) already
showed this behavior under the Terraform-based engine. After migrating this
bundle to the direct deployment engine, every continuous pipeline in the
bundle now exhibits the same behavior — not just that one.
Detailed plan
Can provide a redacted bundle plan -o json -t <target> on request — happy
to capture it if a maintainer confirms this is the most useful next
diagnostic, and to redact accordingly.
Debug Logs
Can provide on request, redacted.
- Dominant language
- Go
- Stars
- 396
- Forks
- 233
- Avg merge
- 2d 52m
- Merged PRs (30d)
- 276
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from databricks/cli
-
DABs
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
databricks/cli#6670 ·
-
DABs PyDABs
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
databricks/cli#3926 · 4 comments ·
-
Difficulty 4/5 3-5 days Newbie friendliness 68/100
databricks/cli#6786 ·
-
Difficulty 4/5 3-5 days Newbie friendliness 45/100
databricks/cli#6785 ·
-
Difficulty 4/5 3-5 days Newbie friendliness 45/100
databricks/cli#6770 ·
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
-
kind/bug needs-triage
Difficulty 1/5 Under an hour Newbie friendliness 72/100
matrixorigin/matrixone#29223 ·
-
needs-acceptance wg/data-plane-networking
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
vllm-project/semantic-router#4024 · 1 comment ·