Add ability to directly reference dependecy libraries from same DAB for Databricks Applications

Open
  • Is something wrong?
#5,094 3 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
45/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Quiet
Tech stack
python
Domain
cli, cloud

Research direction

Start by tracing how application YAML dependencies and bundle job dependencies are handled in databricks-cli, then compare that behavior with the uv workspace configuration shown in the issue. Define the desired deployment order and dependency resolution for an app referencing the same-bundle LIB, including what should replace the package-feed or volume workaround.

Written by the indexing model from the issue text.

Description

DABs Enhancement

I have following setup for bundle:

  • Shared library (LIB) which contain set of pandera DataFrameModels (uv-based dependency management, python) which getting build into whl
  • Job (JOB), which uses DataFrameModels from LIB. Job yaml declares LIB as a dependency
  • Databricks App (APP), which also want to use DataFrameModels from that LIB.

Since I can`t declare app dependencies from bundle itself, I either forced to use package feed or put that LIB into volume, so I can reference it in my requirements/pyproject.toml. It itself creates little bit inconsistency, since then I need somehow handle local development with that dependency or run uv build/run where volume is not available.
However, because of the #1958 I also cannot have that library and volume in one package.

So in such setup, DABs force me to have at least two bundles, first with volume, second - with app and library. In addition to it, I am not sure, will app be deployed first and if I will use lifecycle events (lifecycle.started from #4672 what will happen first - whl gets into the volume, or app will restart/redeploy.

In addition, since JOB is referencing LIB whl as a dependency, it means that it is actually delivered to the Databricks. So its there - in the cloud, but app have no any ability to reference it.

I think if application YAML will support dependency declaration from bundle - it will help.
Other way around - better support uv tooling. In my setup I have uv workspaces in place, so can reference dependency from bundle by using it:

dependencies = [
    "LIB",
]

[tool.uv.sources]
LIB = { workspace = true }

Maybe then databricks-cli can do something about that.

Dominant language
Go
Stars
427
Forks
267
Avg merge
2d 1h
Merged PRs (30d)
233

Getting set up

  • Ships a Dockerfile or Docker Compose file
  • Has a pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from databricks/cli

All issues in databricks/cli

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.