Feature request: native bundle variable substitution in explicitly selected files

Open
  • Is something wrong?
#7,100 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
go
Domain
cli

Research direction

Start by reading the bundle configuration and deployment staging entry points, then trace how target variables are resolved and applied to native bundle configuration. The issue does not name specific files or tests; identify the relevant implementation and tests for file selection, rendering, and staging. Done means an opt-in workflow renders selected files from the resolved target context without modifying sources and covers the requested errors, escaping, sensitive values, and notebook behavior.

Written by the indexing model from the issue text.

Description

Feature request

Support opt-in deployment-time substitution of bundle variables in additional files and directories, using the same resolved context as native bundle configuration.

For example, a bundle could explicitly select SQL migrations, JSON/YAML configuration files, and notebook sources containing ${var.<name>}. The CLI would prepare rendered copies for deployment or for use by an external deployment step.

Example use case

The bundle defines catalog and schema variables, with different values per target. A separate SQL file contains:

CREATE TABLE IF NOT EXISTS `${var.target_catalog}`.`${var.target_schema}`.example (
  id BIGINT
);

A JSON payload contains:

{
  "catalog": "${var.target_catalog}",
  "schema": "${var.target_schema}"
}

We would like to select these files by path, directory, or glob and have Databricks apply native variable resolution when preparing them for the selected target.

Current workaround and cost

We maintain a separate renderer in a deployment wrapper. It gets values from bundle validate -o json, substitutes them into additional files, and restores the working tree afterward.

This duplicates file-rendering and lifecycle work. It also couples the wrapper to validation-output redaction: with UC secrets present, unrelated string variable values are redacted even though native deployment has the actual values internally.

Reading defaults and resolving overrides ourselves would create a second implementation of Databricks bundle semantics. We want the CLI to own resolution.

Requested capability
  • Explicitly opt in files, directories, or globs; ordinary synchronized files should keep their current behavior by default.
  • Use the selected target's native resolution, including shared configuration, existing variable precedence, lookups, and substitutions.
  • Prepare rendered copies without modifying source files in the working tree.
  • Offer the rendered files through deployment staging and/or an output directory usable by external pre/post-deployment steps. A deployment-only upload transform would not cover SQL migrations executed by another tool.
  • Report unresolved references with file locations and provide a way to keep literal placeholder syntax.
  • Define escaping and type behavior for supported formats. For .ipynb, substitute cell source without rewriting outputs or metadata.
  • Define sensitive-value handling so substitutions cannot inadvertently appear in logs or source control; document any restrictions.

The configuration key and command shape are up to the maintainers. The core request is reuse of the native resolved bundle context for explicitly selected file contents.

Benefit

This would let consumers keep a single source of truth for target-specific values, reduce custom renderers and Git restoration logic, and avoid depending on display-output redaction for file preparation.

Native file rendering would solve the payload-preparation portion of our wrapper. External SQL execution would still need a supported way to obtain its non-secret connection settings, such as the warehouse ID.

Related issue

#6000 describes a related JSON/SQL substitution need for Genie. This request is broader: explicitly selected file contents and rendered output that external deployment steps can consume.

If a supported workflow already provides this capability, a pointer to it would be appreciated.

Dominant language
Go
Stars
427
Forks
267
Avg merge
2d 1h
Merged PRs (30d)
233

Getting set up

  • Ships a Dockerfile or Docker Compose file
  • Has a pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from databricks/cli

All issues in databricks/cli

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.