Job compute support (or docs)

Open
  • Is something wrong?
#1,743 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 6 days

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
python
Domain
data-engineering

Research direction

Start by tracing how dbt-databricks submits SQL and how its job compute and submission methods are configured; the issue does not name specific files or tests. Determine whether SQL can run on the job's own compute, then document the technical constraint and compare dbt-spark session mode if support is not feasible. Done means job compute support is implemented, or the requested limitations and tradeoffs are explained in the docs.

Written by the indexing model from the issue text.

Description

enhancement
Describe the feature

Support running dbt-databricks SQL on the job's own job compute, instead of requiring a SQL warehouse or all-purpose cluster.

If that's not possible, please document why, including:

  • The technical reason job compute can't be a target
  • How this compares to dbt-spark method: session, which can run on the current Spark session (there is only a brief mention in the docs)
  • What you'd lose by using dbt-spark session mode instead
  • Why dbt-sparks session method is not compatible with dbt-databricks.
Describe alternatives you've considered
  • dbt-spark method: session: runs on the job cluster, but loses Databricks-specific features and isn't the recommended adapter.
  • Python models with submission_method: job_cluster: Python models only, and creates a new cluster per model.
  • SQL warehouse / all-purpose cluster: works, but adds a second compute resource.
Additional context

I understand the dbt task is an orchestrator, with the job cluster running the dbt CLI and SQL executing elsewhere. But this holds a lot of clients back: they expect their job compute to do the work, and the extra compute adds cost and setup overhead. The docs don't explain why this split is required. And honestly: I'm just tired of having the discussion this doesn't weigh up with the convenience of using dbt lol.

Who will this benefit?
  • Teams running dbt in multi-task jobs who want everything on job compute (e.g. ingest → dbt → notebook).
  • Teams without SQL warehouses, or with restrictions on all-purpose compute.
  • Anyone choosing between dbt-databricks and dbt-spark.
  • Enabling dbt-databricks-factory so users can use native databricks orchestration and restarts. https://github.com/mwojtyczka/databricks-dbt-factory
Are you interested in contributing this feature?

Maybe

Dominant language
Python
Stars
391
Forks
231
Avg merge
9d 7h
Merged PRs (30d)
29

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from databricks/dbt-databricks

All issues in databricks/dbt-databricks

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.