Hacktoberfest 2026：維護者為十月標記出來的 issue，仍然開放、適合新手。 瀏覽 Hacktoberfest issue

`Table.scan(options=...)` silently ignores S3 properties for FileIO during data materialization (`to_pandas` / `to_arrow`)

未關閉
  • 哪裡不對？
#3,166 1 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視

維護者通常 1 天內回覆

還沒有人認領這個 Issue。

  • #3258 來自 @rccreager —— 已關閉，未合併

評估

難度
4/5
預估耗時
3-5 天
新手友好度
55/100
Issue 類型
缺陷
描述清晰度
基本清楚
活躍度
冷清
技術堆疊
aws, python
領域
data-engineering

研究方向

從 pyiceberg/table/init.py 中的 DataScan.to_arrow() 開始，接著追蹤在 to_arrow() 和 to_pandas() 具現化期間 ArrowScan 如何接收 self.io。確認 scan 選項和 FileIO 屬性的處理方式，並考慮現有的 scan／具現化測試入口。完成的定義是，scan 層級的 S3 逾時選項在讀取 manifest 和資料期間會由 FileIO 遵循，並具備回歸測試涵蓋。

由索引模型根據 Issue 內容生成。

描述

Apache Iceberg version

0.11.0 (latest release)

Please describe the bug 🐞

Description:
When passing an options dictionary to Table.scan(options=...), the properties (such as s3.connect-timeout or s3.request-timeout) are accepted by the DataScan object but are never propagated to the underlying FileIO (e.g., PyArrowFileIO) when actual data materialization occurs via methods like to_pandas() or to_arrow().
Because ArrowScan is initialized with the FileIO that was created during catalog instantiation (table.io), any S3-specific configurations provided at the scan level are completely bypassed. This causes operations reading numerous manifest files to fall back to the AWS C++ SDK default timeouts (often 10s-30s), leading to unexpected curlCode: 28 (Timeout was reached) errors even when generous timeouts are explicitly requested in the scan options.

Steps to Reproduce:

# 1. Load catalog with default (or no) S3 timeout properties

from pyiceberg.catalog import load_catalog
catalog = load_catalog("my_catalog", **{
    "uri": "...",
    "s3.endpoint": "..."
})
table = catalog.load_table("my_namespace.my_table")

# 2. Attempt to scan with explicit S3 timeout options

scan_options = {
    "s3.connect-timeout": "600.0",
    "s3.request-timeout": "600.0"
}


# The options are accepted by DataScan...

scan = table.scan(options=scan_options)
# 3. ...but completely ignored during S3 I/O operations (ArrowScan)
# This may throw a timeout error if RGW/S3 latency spikes, ignoring the 600s setting above.
df = scan.to_pandas()
Expected Behavior:

Properties passed via options in Table.scan() should cascade down and either update or override the table.io.properties for the duration of the scan. Specifically, s3.* configurations should be respected by the underlying FileIO (e.g., PyArrowFileIO) when downloading manifest lists or data files.

Actual Behavior:

The options passed to Table.scan() are stored in the DataScan instance but are never passed to the ArrowScan class or the FileIO instance during to_arrow() / to_pandas().
The ArrowScan relies entirely on the unmodified self.io object originally initialized by the catalog:

# In pyiceberg/table/__init__.py -> DataScan.to_arrow()
        return ArrowScan(
            self.table_metadata, 
            self.io,  # <--- options are missing here!
            self.projection(), 
            self.row_filter, 
            self.case_sensitive, 
            self.limit
        ).to_table(self.plan_files())

Environment:

  • PyIceberg Version: 0.11.1 (and earlier)
  • PyArrow Version: 18.0.0
  • Storage: Ceph S3 / Rados Gateway (RGW)
Suggested Fix:

Ideally, DataScan should merge its options with self.io.properties and instantiate a new FileIO, or ArrowScan should be modified to accept the scan-level options and apply them dynamically to the FileSystem instance before reading files.

Willingness to contribute
  • I can contribute a fix for this bug independently
  • I would be willing to contribute a fix for this bug with guidance from the Iceberg community
  • I cannot contribute a fix for this bug at this time
主要語言
Python
星號
1.1k
分支
606
平均合併
1 天 11 小時
30 天內合併 PR
75

環境準備

  • 沒有 Dockerfile 或 Docker Compose 檔案
  • 有 Pull Request 範本
  • 沒有貢獻指南

從這裡開始

  1. 先讀完整個 Issue，再讀專案的貢獻指南。
  2. 在 Issue 下留言說明你要接手 —— 這能避免兩個人做同樣的事。
  3. Fork 儲存庫，在一個分支上完成修改。
  4. 送出 Pull Request，並在描述裡引用這個 Issue 編號。

apache/iceberg-python 的其他 Issue

查看 apache/iceberg-python 的全部 Issue

相似的 Issue

更多 Python Issue

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。