apache/airflow

Explore the possibility of generic `SqlToGcsOperator`

Open

#21,599 opened on Feb 15, 2022

View on GitHub
 (3 comments) (0 reactions) (0 assignees)Python (16,781 forks)batch import
area:providersgood first issuekind:taskprovider:google

Repository metrics

Stars
 (44,809 stars)
PR merge metrics
 (Avg merge 7d 18h) (834 merged PRs in 30d)

Description

Body

Recently SqlToS3Operator was added in PR https://github.com/apache/airflow/pull/20807 (It also deprecated MySqlToS3Operator)

In Google provider Currently we have BaseSQLToGCSOperator with PrestoToGCSOperator, MSSQLToGCSOperator, MySQLToGCSOperator, OracleToGCSOperator, PostgresToGCSOperator, TrinoToGCSOperator, DummySQLToGCSOperator

This task is to check if we can have a generic SqlToGCSOperator just like there is for AWS so we won't need class per each provider...

Committer

  • I acknowledge that I am a maintainer/committer of the Apache Airflow project.

Contributor guide

Research direction
Study the existing BaseSQLToGCSOperator and its subclasses, and the SqlToS3Operator PR #20807 to understand the pattern. Then propose a generic SqlToGcsOperator that can handle multiple SQL dialects via a connection parameter.
Tech stack
python
Domain
backendapiinfrastructure
Issue type
Feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
Half day
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Clear
Prerequisites
PythonApache AirflowGCS
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
55