Job compute support (or docs)

Abierto
  • ¿Algo no está bien?
#1,743 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 6 días

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
35/100
Tipo de issue
Nueva funcionalidad
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
python
Área
data-engineering

Línea de trabajo

Start by tracing how dbt-databricks submits SQL and how its job compute and submission methods are configured; the issue does not name specific files or tests. Determine whether SQL can run on the job's own compute, then document the technical constraint and compare dbt-spark session mode if support is not feasible. Done means job compute support is implemented, or the requested limitations and tradeoffs are explained in the docs.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

enhancement
Describe the feature

Support running dbt-databricks SQL on the job's own job compute, instead of requiring a SQL warehouse or all-purpose cluster.

If that's not possible, please document why, including:

  • The technical reason job compute can't be a target
  • How this compares to dbt-spark method: session, which can run on the current Spark session (there is only a brief mention in the docs)
  • What you'd lose by using dbt-spark session mode instead
  • Why dbt-sparks session method is not compatible with dbt-databricks.
Describe alternatives you've considered
  • dbt-spark method: session: runs on the job cluster, but loses Databricks-specific features and isn't the recommended adapter.
  • Python models with submission_method: job_cluster: Python models only, and creates a new cluster per model.
  • SQL warehouse / all-purpose cluster: works, but adds a second compute resource.
Additional context

I understand the dbt task is an orchestrator, with the job cluster running the dbt CLI and SQL executing elsewhere. But this holds a lot of clients back: they expect their job compute to do the work, and the extra compute adds cost and setup overhead. The docs don't explain why this split is required. And honestly: I'm just tired of having the discussion this doesn't weigh up with the convenience of using dbt lol.

Who will this benefit?
  • Teams running dbt in multi-task jobs who want everything on job compute (e.g. ingest → dbt → notebook).
  • Teams without SQL warehouses, or with restrictions on all-purpose compute.
  • Anyone choosing between dbt-databricks and dbt-spark.
  • Enabling dbt-databricks-factory so users can use native databricks orchestration and restarts. https://github.com/mwojtyczka/databricks-dbt-factory
Are you interested in contributing this feature?

Maybe

Lenguaje dominante
Python
Estrellas
391
Forks
231
Merge medio
9 d 7 h
PR fusionados (30 d)
29

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de databricks/dbt-databricks

Todos los issues de databricks/dbt-databricks

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.