apache/airflow

Extend Databricks Operators to DBFS Interaction

Open

#39,262 opened on Apr 25, 2024

View on GitHub
 (6 comments) (0 reactions) (0 assignees)Python (16,781 forks)batch import
area:providersgood first issuekind:featureprovider:databricks

Repository metrics

Stars
 (44,809 stars)
PR merge metrics
 (Avg merge 7d 18h) (834 merged PRs in 30d)

Description

Description

Create operators and Hook to interact with Databricks' DBFS (https://docs.databricks.com/api/workspace/dbfs)

Use case/motivation

As per latest databricks plugin (https://github.com/apache/airflow/tree/main/airflow/providers/databricks) there is no possibility to interact with DBFS API.

As I had to do it in my job (and I have it quite developed), I thought it'd be a good idea to share it with the community

So far, I've got:

  • An operator that uploads files to DBFS
  • A hook that interacts with the DBFS API, respecting Databricks' Hooks logic and inheriting from BaseDatabricksHook

As part of the PR, I'd add:

  • Some more operators (getting files, getting files metadata, deleting files)
  • Tests in line with Airflow's test suite

Please LMK if you consider this a relevant contribution or not

Related issues

As one of the DBFS API endpoints uses PUT as verb., I'd need to include a modification in BaseDatabricksHook, because it is not supporting PUT ATM (see https://github.com/apache/airflow/blob/main/airflow/providers/databricks/hooks/databricks_base.py#L584)

Are you willing to submit a PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Contributor guide

Research direction
Examine the existing BaseDatabricksHook to understand how to add PUT support, then implement the DBFS operators (upload, get, delete, metadata) and a DBFS hook inheriting from BaseDatabricksHook. Write tests following Airflow's test patterns.
Tech stack
python
Domain
backendapidevops
Issue type
Feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-2 days
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Mostly clear
Prerequisites
PythonGitAirflow
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
50