apache/airflow
GitHub で見るEnable Streaming from SFTP to GCS in `SFTPToGCSOperator`
Open
#34,995 opened on 2023年10月17日
area:providersgood first issuekind:featureprovider:googleprovider:sftp
Repository metrics
- Stars
- (44,809 stars)
- PR merge metrics
- (平均マージ 7d 18h) (30d で 834 merged PRs)
説明
Description
The current SFTPToGCSOperator in Airflow does not support streaming objects directly from SFTP to Google Cloud Storage (GCS). This design poses a challenge since both Composer and Airflow best practices recommend against assigning large amounts of memory to workers. Transferring sizeable files from SFTP to GCS without streaming can potentially max out worker memory.
Use case/motivation
Prevent worker memory exhaustion, especially when dealing with large files.
Related issues
No response
Are you willing to submit a PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct