aws/aws-sdk-pandas

`athena.to_parquet` fails when `mode=overwrite_partitions` and `partition_cols` contains something like `hour(timestamp_col)`.

Open

#2,845 opened on 2024年6月4日

GitHub で見る
 (3 comments) (0 reactions) (0 assignees)Python (3,560 stars) (630 forks)batch import
backlogbughelp wanted

説明

Describe the bug

When using s3.to_parquet to update a parquet file that is partitioned by a time interval or a timestamp "attribute" (such as year, month, hour, etc.), the function fails because for this mode the implementation assumes that the values of partition_cols are names of the parquet / table columns, and it does not find something like hour(column) in the dataframe columns.

I think the problem is this line, which uses the function delete_from_iceberg_table, which expects column names.

How to Reproduce

Expected behavior

I expect the partition_cols option to accept anything that can be used to partition a parquet. In particular, anything that is accepted when the argument mode is append or overwrite instead of overwrite_partitions.

Your project

No response

Screenshots

No response

OS

Ubuntu 22.04

Python version

3.10

AWS SDK for pandas version

3.7.3

Additional context

No response

コントリビューターガイド

技術スタック
pythonpandassql
領域
datacloud
Issue 種別
bug
難度新しい貢献者にとっての実装難度です。1 は非常に小さな変更、5 は専門的な作業です。
3
推定時間経験ある貢献者が調査、実装、テスト、pull request 準備にかけるおおよその時間範囲です。
half day
活動状況Issue が今どれだけ取り組みやすいかを示します。新しい、活発、古い、ブロック中、またはメンテナー入力待ちなどです。
fresh
明確さ期待される変更、受け入れ条件、次の手順がどれだけ明確かを示します。
clear
前提条件
PythonAWS Athenapandaspartitioning concepts
初心者向け度初回貢献者にどれだけ取り組みやすいかを 1-100 で推定したスコアです。
55
調査方針
Examine line 452 in awswrangler/athena/ write iceberg.py. Understand how delete from iceberg table uses partition cols. The fix should parse expressions like hour(timestamp col) to extract the column name and apply the transformation when partitioning. Check existing tests for overwrite partitions mode and add a test case with expression based partitions.