Validation Failing with`experimental.immutable_folder: true` Despite Plan and Deploy Succeeding

Start at bundle/config/mutator/override_immutable_folder.go and bundle/config/validate/folder_permissions.go, then find where the validate pipeline is assembled versus the plan/deploy pipeline (which runs the snapshot.Upload mutator that creates internal_immutable_snapshots.immutable). Determine why the resource reference stays a literal ${...} during validate. Done: databricks bundle validate no longer sends the unresolved snapshot path to GET /api/2.0/workspace/get-status, covered by a test around the validate mutator.

Describe the issue

bundle validate fails when experimental.immutable_folder: true is set, even though bundle plan and bundle deploy seem to work correctly.

The OverrideImmutableFolder mutator rewrites workspace.artifact_path and workspace.file_path to reference ${resources.internal_immutable_snapshots.immutable.full_path}. During plan and deploy, the internal_immutable_snapshots.immutable resource is created and the reference resolves correctly. During validate, however, the resource is never created - the reference passes through as a literal string, and the validate:folder_permissions mutator sends it to the workspace API, which rejects it.

Configuration

Minimal databricks.yml to reproduce:

bundle:
  name: my.bundle
  engine: direct

experimental:
  immutable_folder: true

workspace:
  root_path: /Workspace/Bundles/my/bundle

targets:
  dev:
    workspace:
      host: https://<workspace-url>
    run_as:
      service_principal_name: <service-principal-id>
    permissions:
      - level: CAN_MANAGE
        service_principal_name: <service-principal-id>

There are also a few basic example jobs and SDP pipelines that point to notebooks using relative paths e.g. ../notebooks/hello_world.ipynb.

Note: I was testing using a pre-existing bundle that was already deployed using the direct engine prior to enabling immutable_folder. I did not test with a brand-new (never-deployed-before) bundle. Tested once with a custom workspace.root_path and once with the default and got the same result both times.

Steps to reproduce the behavior

1. Set experimental.immutable_folder: true in databricks.yml
2. Run databricks bundle validate --target dev (as a service principal)
3. See error:

   Error: Path (${resources.internal_immutable_snapshots.immutable.full_path}/artifacts) doesn't start with '/'
   Error: Path (${resources.internal_immutable_snapshots.immutable.full_path}/files) doesn't start with '/'
   

Note: bundle plan and bundle deploy both succeed — the internal_immutable_snapshots.immutable resource is created and the snapshot path resolves correctly. Only validate is affected.

Expected Behavior

Unsure. I assume bundle validate should either resolve the internal_immutable_snapshots.immutable resource (as plan and deploy do), or the validate:folder_permissions step should skip validation for paths that reference immutable snapshot resources that are only created at deploy time.

Actual Behavior

The mutator runs and successfully rewrites:

• workspace.artifact_path → ${resources.internal_immutable_snapshots.immutable.full_path}/artifacts
• workspace.file_path → ${resources.internal_immutable_snapshots.immutable.full_path}/files

However, during validate, the internal_immutable_snapshots.immutable resource is never created (it is absent from bundle validate --output json). The variable reference passes through as the literal string ${resources.internal_immutable_snapshots.immutable.full_path}, and validate:folder_permissions sends it to the GET /api/2.0/workspace/get-status endpoint, which rejects it because it doesn't start with /.

In contrast, plan and deploy both create the resource successfully:

• plan output shows recreate internal_immutable_snapshots.immutable and the snapshot.Upload mutator resolves the path to /Workspace/Users/<sp-id>/.snapshots/<bundle-hash>/<deployment-hash>
• deploy output shows Recreated internal_immutable_snapshots.immutable and creates the snapshot via POST /api/2.0/repos/snapshots, with job notebook paths correctly pointing to the snapshot folder

OS and CLI version

• CLI version: v1.19.0 (latest)
• OS: macOS 26.6

Is this a regression?

Unknown - this is my first time testing experimental.immutable_folder. I have not tested with any prior CLI version.

Detailed plan

bundle plan output:

Building my_bundle_custom_lib_a...
recreate internal_immutable_snapshots.immutable
update jobs.example_job
update jobs.sp_rights_analysis
update pipelines.example_pipeline

Plan: 1 to add, 3 to change, 1 to delete, 10 unchanged

Filtered debug output of bundle plan --log-level DEBUG -t dev 2>&1 | grep -i "immutable\|snapshot":

18:07:42 Debug: Apply pid=81169 mutator=OverrideImmutableFolder
18:07:49 Debug: Apply pid=81169 mutator=snapshot.Upload
18:07:49 Debug: GET /api/2.0/repos/snapshots/rootpath
<   "path": "/Workspace/Users/<sp-id>/.snapshots/"
< } pid=81169 mutator=snapshot.Upload sdk=true
18:07:49 Debug: GET /api/2.0/repos/snapshots/rootpath
<   "path": "/Workspace/Users/<sp-id>/.snapshots/"
18:07:49 Debug: GET /api/2.0/workspace/get-status?path=/Workspace/Users/<sp-id>/.snapshots/<bundle-hash>/<deployment-hash>
<   "path": "/Workspace/Users/<sp-id>/.snapshots/<bundle-hash>/<deployment-hash>",
recreate internal_immutable_snapshots.immutable

Redacted bundle plan -o json (trimmed to the immutable snapshot resource and one job showing the dependency – other resources omitted for brevity):

{
  "plan_version": 2,
  "cli_version": "1.19.0",
  "plan": {
    "resources.internal_immutable_snapshots.immutable": {
      "action": "recreate",
      "new_state": {
        "value": {
          "relative_path": "<bundle-hash>/<new-deployment-hash>",
          "full_path": "/Workspace/Users/<deployer-sp-id>/.snapshots/<bundle-hash>/<new-deployment-hash>",
          "bundle_id": "<bundle-hash>",
          "acl": [
            { "user_name": "<run-as-sp-id>", "permission_level": "CAN_READ" },
            { "service_principal_name": "<run-as-sp-id>", "permission_level": "CAN_READ" },
            { "user_name": "<user-email>", "permission_level": "CAN_READ" }
          ],
          "zip_path": "<local-path>/.databricks/bundle/dev/snapshots/<new-deployment-hash>.zip"
        }
      },
      "remote_state": {
        "relative_path": "<bundle-hash>/<old-deployment-hash>",
        "full_path": "/Workspace/Users/<deployer-sp-id>/.snapshots/<bundle-hash>/<old-deployment-hash>"
      },
      "changes": {
        "full_path": {
          "action": "recreate",
          "reason": "immutable",
          "old": "/Workspace/Users/<deployer-sp-id>/.snapshots/<bundle-hash>/<old-deployment-hash>",
          "new": "/Workspace/Users/<deployer-sp-id>/.snapshots/<bundle-hash>/<new-deployment-hash>",
          "remote": "/Workspace/Users/<deployer-sp-id>/.snapshots/<bundle-hash>/<old-deployment-hash>"
        },
        "relative_path": {
          "action": "recreate",
          "reason": "immutable"
        },
        "zip_path": {
          "action": "recreate",
          "reason": "immutable"
        }
      }
    },
    "resources.jobs.example_job": {
      "depends_on": [
        {
          "node": "resources.internal_immutable_snapshots.immutable",
          "label": "${resources.internal_immutable_snapshots.immutable.full_path}"
        }
      ],
      "action": "update",
      "new_state": {
        "value": {
          "tasks": [
            {
              "notebook_task": {
                "notebook_path": "/Workspace/Users/<deployer-sp-id>/.snapshots/<bundle-hash>/<new-deployment-hash>/files/src/main/databricks/notebooks/hello_world"
              },
              "task_key": "serverless_task"
            }
          ]
        }
      },
      "changes": {
        "tasks[task_key='serverless_task'].notebook_task.notebook_path": {
          "action": "update",
          "old": "/Workspace/Users/<deployer-sp-id>/.snapshots/<bundle-hash>/<old-deployment-hash>/files/src/main/databricks/notebooks/hello_world",
          "new": "/Workspace/Users/<deployer-sp-id>/.snapshots/<bundle-hash>/<new-deployment-hash>/files/src/main/databricks/notebooks/hello_world",
          "remote": "/Workspace/Users/<deployer-sp-id>/.snapshots/<bundle-hash>/<old-deployment-hash>/files/src/main/databricks/notebooks/hello_world"
        }
      }
    }
  }
}

Key observations from the plan:

• plan correctly creates the internal_immutable_snapshots.immutable resource with action: "recreate" and reason: "immutable", resolving full_path to a real snapshot path
• Jobs depend on resources.internal_immutable_snapshots.immutable and their notebook_path values update to the new snapshot hash
• plan runs the snapshot.Upload mutator (which calls GET /api/2.0/repos/snapshots/rootpath), but validate does not – it only runs OverrideImmutableFolder

Debug Logs

bundle validate (fails)

databricks bundle validate -t dev
Error: Path (${resources.internal_immutable_snapshots.immutable.full_path}/artifacts) doesn't start with '/' (400 INVALID_PARAMETER_VALUE)

Endpoint: GET https://<workspace-url>/api/2.0/workspace/get-status?path=%24%7Bresources.internal_immutable_snapshots.immutable.full_path%7D%2Fartifacts
HTTP Status: 400 Bad Request
API error_code: INVALID_PARAMETER_VALUE
API message: Path (${resources.internal_immutable_snapshots.immutable.full_path}/artifacts) doesn't start with '/'

Error: Path (${resources.internal_immutable_snapshots.immutable.full_path}/files) doesn't start with '/' (400 INVALID_PARAMETER_VALUE)

Endpoint: GET https://<workspace-url>/api/2.0/workspace/get-status?path=%24%7Bresources.internal_immutable_snapshots.immutable.full_path%7D%2Ffiles
HTTP Status: 400 Bad Request
API error_code: INVALID_PARAMETER_VALUE
API message: Path (${resources.internal_immutable_snapshots.immutable.full_path}/files) doesn't start with '/'

Name: my.bundle
Target: dev
Workspace:
  Host: https://<workspace-url>
  User: <sp-id>
  Path: /Workspace/Bundles/my/bundle

Found 2 errors

Filtered debug output (bundle validate --log-level DEBUG -t dev 2>&1 | grep -i "immutable\|snapshot"):

18:06:23 Debug: Apply pid=80390 mutator=OverrideImmutableFolder
18:06:24 Debug: GET /api/2.0/workspace/get-status?path=${resources.internal_immutable_snapshots.immutable.full_path}/files
<   "message": "Path (${resources.internal_immutable_snapshots.immutable.full_path}/files) doesn't start with '/... (1 more bytes)"
18:06:24 Debug: non-retriable error: Path (${resources.internal_immutable_snapshots.immutable.full_path}/files) doesn't start with '/' pid=80390 mutator=validate:folder_permissions sdk=true
18:06:24 Debug: GET /api/2.0/workspace/get-status?path=${resources.internal_immutable_snapshots.immutable.full_path}/artifacts
<   "message": "Path (${resources.internal_immutable_snapshots.immutable.full_path}/artifacts) doesn't start wit... (5 more bytes)"
18:06:24 Debug: non-retriable error: Path (${resources.internal_immutable_snapshots.immutable.full_path}/artifacts) doesn't start with '/' pid=80390 mutator=validate:folder_permissions sdk=true

bundle deploy (succeeds)

databricks bundle deploy -t dev
Building my_bundle_custom_lib_a...
Recreated internal_immutable_snapshots.immutable
Updated jobs.sp_rights_analysis
Updated jobs.example_job
Updated pipelines.example_pipeline
Files: 0 uploaded, 0 deleted
Resources: 1 created, 3 changed, 1 deleted, 10 unchanged

Filtered debug output (bundle deploy --log-level DEBUG -t dev 2>&1 | grep -i "immutable\|snapshot") confirms the snapshot.Upload mutator runs and the resource is created via POST /api/2.0/repos/snapshots, with notebook paths correctly resolving to /Workspace/Users/<sp-id>/.snapshots/<bundle-hash>/<snapshot-hash>/files/....