Read Parquet exceptions thrown with PyArrow S3FileSystem
还没有人认领这个 Issue。
评估
调研方向
从 dask_expr/io/parquet.py 开始，重点检查 ReadParquetPyarrowFS.normalized_path 和 _normalize_and_strip_protocol，并使用提供的 glob 和 list 示例。根据 traceback 验证行为，然后添加或更新相关测试，使 glob 路径和可迭代路径能够在 pyarrow.fs.S3FileSystem 中正常工作且不会出现递归错误。
由索引模型根据 Issue 内容生成。
描述
Describe the issue:
Hi team,
We are intensive users of Dask and it's a great product!
We use Apache Arrow's pyarrow.fs.S3FileSystem in our ecosystem rather than s3fs.S3FileSystem due to performance and deadlock issues that were found during multiprocessing.
We're able to retrieve single files and entire directories of many files successfully. But using either a glob path or a interable of paths the read_parquet API throws exceptions with maximum recursion depth.
If would really helpful if the team can investigate? Many thanks!
Minimal Verifiable Example:
import dask.dataframe
import pyarrow.fs
# pyarrow s3 file system client
arrow_s3 = pyarrow.fs.S3FileSystem(
access_key=s3_tokens["AccessKeyId"],
session_token=s3_tokens["SessionToken"],
secret_key=s3_tokens["SecretAccessKey"],
region=_REGION,
scheme="http",
)
# works as expected for both target_1 and target_2
df = dask.dataframe.read_parquet(
path=f"s3://{bucket}/{key}/target_1.parquet",
filesystem=arrow_s3,
)
# works as expected for entire folder, numerous parquet files
df = dask.dataframe.read_parquet(
path=f"s3://{bucket}/{key}",
filesystem=arrow_s3,
)
df
# glob path raises exception
df = dask.dataframe.read_parquet(
path=f"s3://{bucket}/{key}/*.parquet",
filesystem=arrow_s3,
)
# iterable of paths, either tuple or list, raises exception
df = dask.dataframe.read_parquet(
path=[
f"s3://{bucket}/{key}/target_1.parquet",
f"s3://{bucket}/{key}/target_2.parquet",
],
filesystem=arrow_s3,
)
Exception:
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
File /env/jupyter/venv/lib/python3.12/site-packages/dask_expr/_core.py:446, in Expr.__getattr__(self, key)
445 try:
--> 446 return object.__getattribute__(self, key)
447 except AttributeError as err:
File /usr/local/lib/python3.12/functools.py:995, in cached_property.__get__(self, instance, owner)
994 if val is _NOT_FOUND:
--> 995 val = self.func(instance)
996 try:
File /env/jupyter/venv/lib/python3.12/site-packages/dask_expr/io/parquet.py:716, in ReadParquetPyarrowFS.normalized_path(self)
714 @cached_property
715 def normalized_path(self):
--> 716 return _normalize_and_strip_protocol(self.path)
File /env/jupyter/venv/lib/python3.12/site-packages/dask_expr/io/parquet.py:1658, in _normalize_and_strip_protocol(path)
1657 for sep in protocol_separators:
-> 1658 split = path.split(sep, 1)
1659 if len(split) > 1:
AttributeError: 'list' object has no attribute 'split'
During handling of the above exception, another exception occurred:
...
RecursionError: maximum recursion depth exceeded while calling a Python object
Normalization failed: type=AttributeError args=<unknown>
Environment:
OS: Amazon Linux release 2 (Karoo)
Linux: 4.14.336-257.566.amzn2.x86_64
Python: 3.12.2
Packages:
arrow: 1.3.0
dask: 2024.3.1
dask-expr: 1.0.4
numpy: 1.26.4
pandas: 2.2.1
pyarrow: 15.0.2
pyarrow-hotfix: 0.6
Install method pip
- 主要语言
- Python
- 星标
- 89
- 派生
- 26
- PR 合并指标
- 30 天内没有已合并 PR
环境准备
- 没有 Dockerfile 或 Docker Compose 文件
- 没有 Pull Request 模板
- 阅读贡献指南
从这里开始
- 先读完整个 Issue，再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库，在一个分支上完成修改。
- 提交 Pull Request，并在描述里引用这个 Issue 编号。
dask/dask-expr 的其他 Issue
-
难度 4/5 3-5 天 新手友好度 38/100
-
难度 3/5 1-2 天 新手友好度 45/100
-
难度 5/5 一周以上 新手友好度 30/100
-
难度 4/5 3-5 天 新手友好度 30/100
-
难度 3/5 1-2 天 新手友好度 25/100
相似的 Issue
-
难度 2/5 1-3 小时 新手友好度 88/100
BasedHardware/omi#20271 ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 92/100
openai/openai-cookbook#3153 ·
维护者通常 1 天内回复
-
cvss-severity:high devguard l3montree-cybersecurity/devguard/devguard pkg:golang/github.com/l3montree-dev/devguard risk:low state:open
难度 2/5 1-3 小时 新手友好度 65/100
l3montree-dev/devguard#3146 · 1 条评论 ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 76/100
-
bug confirmed issue
难度 2/5 1-3 小时 新手友好度 76/100
open-webui/open-webui#31849 · 2 条评论 ·
维护者通常 1 天内回复