pola-rs/polars

Add add_filename to pl.read_csv (and read operations others)

Open

#19,266 创建于 2024年10月16日

在 GitHub 查看
 (6 评论) (3 反应) (0 负责人)Rust (38,496 star) (2,826 fork)batch import
enhancementgood first issue

描述

Description

It is ocasionaly true that the filename of a data file is fairly critical information

Illustratively

Users/
   Alice.csv
   Bob.csv
   Charlie.csv

When using glob patterns to read this data, the file name itself is lost - which all but forces the user to loop over the files and read them manually.


# This does not preserve what row is for what user
df = pl.read_csv('Users/*.csv') 

 # This is a bit long
df = (pl.concat([
                pl.read_csv(file).with_columns(filename=pl.lit(file) 
                for file in glob('Users/*.csv')
            ])
        )

A parameter to add a column with the specific file name when reading data via a glob pattern would be a nice to have.

贡献者指南

技术栈
pythonrust
领域
data
议题类型
feature
难度面向新贡献者的预计实现难度，1 表示很小改动，5 表示专家级工作。
3
预计时间有经验贡献者完成调查、实现、测试并准备 pull request 的粗略时间范围。
3-5 days
活动状态议题当前的可参与程度：新鲜、活跃、陈旧、阻塞或等待维护者输入。
fresh
清晰度议题是否清楚说明期望改动、验收标准和下一步。
clear
前置要求
Polars Python APIRust FFI (pyo3)
新手友好度1-100 的估计分数，表示该议题对首次贡献者的友好程度。
50
研究方向
The implementation involves adding a `add filename` parameter to `pl.read csv` (and other read functions like `read parquet`) in the Python API. This requires modifying the Python function signatures in `polars/py polars/polars/io/csv.py` and passing the parameter through to the Rust backend in `polars/polars io/src/csv.rs`. The column should contain the source filename or path. Review existing patterns like `include filepaths` or similar parameters in the codebase. Discussion in the issue comments may provide further context on desired behavior for multiple files vs single file.