aws/aws-sdk-pandas

Wrangler to support Hudi/Iceberg datasets read/write

Open

Aperta il 22 lug 2022

Vedi su GitHub
 (8 commenti) (11 reazioni) (0 assegnatari)Python (3560 star) (630 fork)batch import
backlogenhancementhelp wanted

Descrizione

Is your idea related to a problem? Please describe. No

Describe the solution you'd like It would be good to have support for CDC data lake formats like Apache Hudi, Apache Iceberg or Detla Lake format.

P.S. Please do not attach files as it's considered a security risk. Add code snippets directly in the message body as much as possible.

Guida contributor

Tech stack
pythonspark
Dominio
databackend
Tipo issue
feature
DifficoltàQuanto dovrebbe essere impegnativa per un nuovo contributor.
5
Tempo stimatoTempo stimato per completare e verificare uno scope piccolo.
over 1 week
Stato attivitàQuanto è probabile che la issue sia ancora attiva e reviewable.
stale
ChiarezzaQuanto chiaramente la issue descrive problema, scope e risultato atteso.
mostly clear
Prerequisiti
PythonAWS Glue/Athena conceptsApache Hudi/Iceberg basics
Adatta ai principiantiQuanto la issue è adatta a contributor alla prima esperienza.
10
Direzione di ricerca
Investigate existing data source implementations in the repository (e.g., S3, Parquet readers) under the `awswrangler` directory. Review the AWS Glue and Athena documentation for native support of Hudi/Iceberg. Examine the closed PRs and discussions for prior attempts. The issue has 8 comments; review them for specific requirements and constraints. Consider starting with a minimal read only implementation to gauge complexity.