area:providersgood first issuekind:featureprovider:googleprovider:sftp
仓库指标
- 星标
- (44,809 个星标)
- PR 合并指标
- (PR 指标待抓取)
描述
Description
The current SFTPToGCSOperator in Airflow does not support streaming objects directly from SFTP to Google Cloud Storage (GCS). This design poses a challenge since both Composer and Airflow best practices recommend against assigning large amounts of memory to workers. Transferring sizeable files from SFTP to GCS without streaming can potentially max out worker memory.
Use case/motivation
Prevent worker memory exhaustion, especially when dealing with large files.
Related issues
No response
Are you willing to submit a PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct