apache/airflow

cluster config generation using cluster config YAML file.

Open

#17.089 geöffnet am 19. Juli 2021

Auf GitHub ansehen
 (8 Kommentare) (0 Reaktionen) (0 zugewiesene Personen)Python (16.781 Forks)batch import
good first issuekind:featureprovider:google

Repository-Metriken

Stars
 (44.809 Stars)
PR-Merge-Metriken
 (Durchschn. Merge 7T 18h) (834 gemergte PRs in 30 T)

Beschreibung

Description

Add cluster config generation from yaml file which is generated using gcloud dataproc clusters export command.

Use case / motivation

We are generating quite a lot of clusters using a cluster config which keeps on changing depending upon project to project. I wrote custom operator which reads yaml config and converts it to dict and then uses the same config to genreate cluster. Wondering if it will be useful if cluster config generator operator can accept a yaml config or can create a new operator.

Are you willing to submit a PR?

I do have a working code for converting yaml stored on gcs to dict.

Related Issues

There is no issue as such, it is a functionality which allows users to specify cluster config as yaml file.

Contributor Guide