apache/airflow

Enable Streaming from SFTP to GCS in `SFTPToGCSOperator`

Open

#34,995 opened on Oct 17, 2023

View on GitHub
 (9 comments) (0 reactions) (0 assignees)Python (16,781 forks)batch import
area:providersgood first issuekind:featureprovider:googleprovider:sftp

Repository metrics

Stars
 (44,809 stars)
PR merge metrics
 (Avg merge 7d 18h) (834 merged PRs in 30d)

Description

Description

The current SFTPToGCSOperator in Airflow does not support streaming objects directly from SFTP to Google Cloud Storage (GCS). This design poses a challenge since both Composer and Airflow best practices recommend against assigning large amounts of memory to workers. Transferring sizeable files from SFTP to GCS without streaming can potentially max out worker memory.

Use case/motivation

Prevent worker memory exhaustion, especially when dealing with large files.

Related issues

No response

Are you willing to submit a PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Contributor guide

Research direction
Explore existing streaming patterns in Airflow for other operators (e.g., S3ToGCSOperator) and adapt them for SFTP. Research the Paramiko library's SFTP file handling capabilities and Google Cloud Storage streaming uploads. Consider using file like objects to stream data without intermediate disk storage.
Tech stack
pythongcp
Domain
backend
Issue type
Feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-2 days
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Clear
Prerequisites
PythonApache AirflowGoogle Cloud Storage
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
70