bundle run on an app fails with "not in RUNNING state" when the app's compute is UPDATING
Maintainers usually reply within 1 day
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 76/100
Research direction
Start with bundle/run/app.go, especially appRunner.Run, which the issue identifies as the failing state check. Trace how it handles compute states and how WaitGetAppActive responds when compute stops during startup. Done when bundle run <app> waits through transient states and handles a STARTING compute that returns to STOPPED, with tests covering these cases.
Written by the indexing model from the issue text.
Description
Describe the issue
databricks bundle run <app> can fail on a newly created app because it checks the app's compute state only once. It starts a compute that is STOPPED or ERROR and waits on one that is STARTING. In any other state, such as UPDATING, it tries to deploy straight away and fails with "not in RUNNING state". The relevant code is bundle/run/app.go, appRunner.Run. It is the same in v1.16.1 and v1.25.0.
There is a related case in the same code. A new app's compute can go from STARTING back to STOPPED on its own. If bundle run reads it during STARTING, it waits for ACTIVE (WaitGetAppActive), and that wait fails when the compute stops instead.
Steps to reproduce the behavior
The failure is intermittent, because it depends on the compute's state at the moment bundle run reads it.
- Define an app in a bundle, with a job that runs before the app is started.
- Run
databricks bundle deploy -t <target>, which creates the app. - Run the job with
databricks bundle run <job> -t <target>. In our case it took about 50 seconds. - Run
databricks bundle run <app> -t <target>. - Sometimes the compute is UPDATING at step 4, and the command fails. In our case this happened in 1 of 4 deploys of a new app.
Polling databricks apps get <app> -o json every 3 seconds during a new-app deploy showed the compute go from STARTING to STOPPED about 20 seconds after bundle deploy created the app.
Expected Behavior
bundle run <app> waits until the compute settles in ACTIVE, STOPPED or ERROR before it starts or deploys the app. It should tolerate transient states such as UPDATING and STOPPING, and a STARTING compute that ends STOPPED.
Actual Behavior
✓ Getting the status of the app my-app
✓ App is in UNAVAILABLE state
✓ App compute is in UPDATING state
Error: Cannot deploy app my-app as it is not in RUNNING state. Please start the app first.
OS and CLI version
Databricks CLI v1.16.1 on Linux, run non-interactively in a deployment pipeline as a service principal, against an AWS workspace. The code path is unchanged in v1.25.0.
Is this a regression?
Not known. We have only run v1.16.1. bundle/run/app.go is the same in v1.16.1 and v1.25.0.
Debug Logs
We don't have a debug-level log of a failing run. The failure is intermittent, and it happened in an unattended pipeline. The output above is the command's normal output, with the app name redacted.
Workaround we use: before bundle run <app>, poll databricks apps get <app> -o json until .compute_status.state is ACTIVE, STOPPED or ERROR, or the status is absent, with a time limit. Then run bundle run <app>.
- Dominant language
- Go
- Stars
- 427
- Forks
- 267
- Avg merge
- 2d 1h
- Merged PRs (30d)
- 233
Getting set up
- Ships a Dockerfile or Docker Compose file
- Has a pull request template
- No contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from databricks/cli
-
Apps commands only accept relative path to app.ymlPossibly taken @yuri-rod claimed this 38 days ago. OpenCLI
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
databricks/cli#6910 ·
Maintainers usually reply within 1 day
-
DABs
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
databricks/cli#6670 · 1 reaction ·
Maintainers usually reply within 1 day
-
Bump vulnerable dependencies workflow is failingPossibly taken @shreyas-goenka claimed this today. Openbump-vuln-deps-failure
databricks/cli#7531 · 1 comment · 7 assignees ·
Maintainers usually reply within 1 day
-
Bundle deploy of an AI/BI dashboard breaks viewers' private bookmarks (UI save + publish does not)Open
Difficulty 4/5 3-5 days Newbie friendliness 45/100
databricks/cli#7419 ·
Maintainers usually reply within 1 day
-
Difficulty 3/5 1-2 days Newbie friendliness 66/100
databricks/cli#7387 · 3 comments ·
Maintainers usually reply within 1 day
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
Maintainers usually reply within 1 day
-
kind/features
Difficulty 2/5 1-3 hours Newbie friendliness 73/100
TencentBlueKing/bk-nodemgr#3991 ·
Maintainers usually reply within 1 day
-
meta
Difficulty 2/5 1-3 hours Newbie friendliness 62/100
last9/last9-mcp-server#554 ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 85/100
-
Plain-http federation signing external_url fails startup with "invalid federation request signature"Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day