Expose per-file write metadata from DataFrame.write_parquet()
还没有人认领这个 Issue。
评估
- 难度
- 4/5
- 预计耗时
- 3-5 天
- 新手友好度
- 48/100
- Issue 类型
- 功能
- 描述清晰度
- 基本清楚
- 活跃度
- 冷清
调研方向
先检查 apache/datafusion#23656 是否已经合并,然后阅读 DataFrame.write_parquet() 的 Python binding,以及链接的 Rust core issue 和 pull request。API 的形式仍未确定:issue 建议直接返回 metadata,或通过 WriteResult 返回。完成的标准是通过 bindings 暴露每个文件的路径、行数和字节大小;序列化的 metadata 是可选的。
由索引模型根据 Issue 内容生成。
描述
Is your feature request related to a problem or challenge?
DataFrame.write_parquet() currently returns None. After writing, there is no way to retrieve per-file metadata (row counts, byte sizes, column statistics) for the files that were produced. This forces consumers that need file-level statistics — such as Apache Iceberg, Delta Lake, and Apache Hudi — to either:
- Re-read Parquet footers from object storage after writing (extra I/O round-trips)
- Bypass DataFusion's write pipeline entirely and use PyArrow's
ParquetWriterwithmetadata_collector
This is a blocker for building a complete DataFusion-based write backend for table formats that require per-file column statistics in their commit metadata (e.g., Iceberg's DataFile entries need column_sizes, null_counts, lower_bounds, upper_bounds, split_offsets).
Describe the solution you'd like
After apache/datafusion#23472 / apache/datafusion#23656 lands in the Rust core, ParquetSink will expose a file_metadata() method returning per-file path, row count, and byte size. The Python bindings should surface this:
# Option A: write_parquet returns metadata directly
metadata = df.write_parquet("/path/to/output/")
# metadata: list[dict] = [
# {"path": "part-0.parquet", "row_count": 500, "byte_size": 4096},
# {"path": "part-1.parquet", "row_count": 500, "byte_size": 3840},
# ]
# Option B: write_parquet returns a WriteResult object
result = df.write_parquet("/path/to/output/")
result.count # 1000
result.file_metadata # list of per-file metadata dicts
At minimum, each file metadata entry should include:
path(str): Object-store path of the written filerow_count(int): Number of rows in this filebyte_size(int): Sum of compressed row group sizes
Optionally (for full table-format integration):
metadata(bytes | None): Serialized ParquetFileMetaData(Thrift compact), enabling consumers to extract column statistics without re-reading the file
Describe alternatives you've considered
- Return just the count (status quo): Insufficient for table format integration.
- Expose via a separate accessor: e.g.
ctx.last_write_metadata()— awkward API, not composable. - Return raw bytes of the full Parquet footer: Maximally informative but heavier. A structured dict with optional raw bytes is more ergonomic.
Additional context
- Upstream dependency: apache/datafusion#23656 adds
DataSink::file_metadata()to the Rust core. This issue tracks exposing it through the Python bindings. - Motivation: PyIceberg is building a pluggable execution backend with DataFusion for bounded-memory operations. A DataFusion write backend would enable single-pass Copy-on-Write deletes (read → filter → write entirely in Rust with spill-to-disk), but requires per-file metadata to construct Iceberg
DataFilecommit entries. - Related: #1624 (per-session object store config) is the other piece needed for a complete DataFusion write backend in PyIceberg.
- 主要语言
- Python
- 星标
- 605
- 派生
- 176
- 平均合并
- 1 天 23 小时
- 30 天内合并 PR
- 8
贡献指南
这个仓库没有索引到贡献指南
从这里开始
- 先读完整个 Issue,再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库,在一个分支上完成修改。
- 提交 Pull Request,并在描述里引用这个 Issue 编号。
apache/datafusion-python 的其他 Issue
-
documentation
难度 2/5 1-3 小时 新手友好度 72/100
apache/datafusion-python#1726 ·
-
难度 2/5 半天 新手友好度 88/100
apache/datafusion-python#1691 ·
-
bug
难度 2/5 1-3 小时 新手友好度 78/100
apache/datafusion-python#1644 ·
-
enhancement
难度 5/5 一周以上 新手友好度 30/100
apache/datafusion-python#1737 ·
-
难度 3/5 1-2 天 新手友好度 76/100
apache/datafusion-python#1735 · 1 条评论 ·
查看 apache/datafusion-python 的全部 Issue
相似的 Issue
-
Add: hunch 未关闭
难度 2/5 1-3 小时 新手友好度 74/100
AbdelStark/awesome-typesafe#104 ·
-
enhancement
难度 2/5 1-3 小时 新手友好度 72/100
-
难度 2/5 1-3 小时 新手友好度 68/100
DiamondLightSource/dodal#2211 ·
-
难度 2/5 1-3 小时 新手友好度 88/100
openml/openml-python#1749 ·
-
难度 2/5 1-3 小时 新手友好度 88/100
sipyourdrink-ltd/bernstein#6191 ·