Hacktoberfest 2026: as issues que os mantenedores marcaram para outubro, abertas e boas para iniciantes. Ver issues do Hacktoberfest

PyArrowFileIO: every small S3 write is a 3-request multipart upload; expose allow_delayed_open

Aberta Para iniciantes
  • Algo errado?
#4,093 0 comentários 0 reações 0 responsáveis Ver no GitHub

Mantenedores costumam responder em até 1 dia

Ninguém assumiu esta issue ainda.

Avaliação

Dificuldade
2/5
Tempo estimado
1-3 horas
Facilidade para iniciantes
70/100
Tipo de issue
Funcionalidade
Clareza
Claramente especificada
Status de atividade
Ativa
Stack de tecnologia
aws, python
Domínio
cloud, data

Direção de pesquisa

Comece em _initialize_s3_fs no PyArrowFileIO, onde o sistema de arquivos S3 é construído a partir de um conjunto fixo de propriedades. Veja como as propriedades s3.* existentes são lidas e repassadas. Adicione a propriedade s3.allow-delayed-open, passe allow_delayed_open somente quando o pyarrow instalado for a versão 21 ou posterior e use true como padrão, conforme proposto na issue. Está pronto quando gravações pequenas são enviadas como um único PutObject e a opção fica ausente com pyarrow mais antigo.

Escrita pelo modelo de indexação a partir do texto da issue.

Descrição

Apache Iceberg version

0.12.0 (also on main)

Please describe the bug 🐞

PyArrowFileIO writes every S3 object as a multipart upload, however small it is. pyarrow's S3FileSystem.open_output_stream starts a multipart upload as soon as the stream opens (apache/arrow#51029). Each metadata.json, manifest list, manifest, version-hint.text and small data file therefore costs three requests: CreateMultipartUpload, UploadPart and CompleteMultipartUpload. All three are billed as writes, even for a 1-byte object.

pyarrow's S3FileSystem has an option for this, allow_delayed_open, since pyarrow 21. With it set, a stream that closes before reaching a part's size is sent as a single PutObject, and a larger one is still a multipart upload. _initialize_s3_fs builds the filesystem from a fixed set of properties, though, so there's no way to set the option through FileIO properties.

The cost is real for a table that commits often. Each commit writes several small metadata objects, so it makes about three times the write requests it needs.

Repro

Run against any S3-compatible endpoint (this was run against a local rustfs):

import pyarrow.fs as fs
from pyiceberg.io.pyarrow import PyArrowFileIO

fs.initialize_s3(fs.S3LogLevel.Debug)  # logs each request

io = PyArrowFileIO({
    "s3.endpoint": "http://127.0.0.1:9000",
    "s3.access-key-id": "...",
    "s3.secret-access-key": "...",
    "s3.region": "us-east-1",
})
with io.new_output("s3://bucket/version-hint.text").create(overwrite=True) as f:
    f.write(b"1")

The debug log shows three requests for the one byte:

POST /bucket/version-hint.text?uploads
PUT  /bucket/version-hint.text?partNumber=1&uploadId=...
POST /bucket/version-hint.text?uploadId=...

When the same write goes through an S3FileSystem built with allow_delayed_open=True, the log shows a single PUT /bucket/version-hint.text.

Proposal

Pass allow_delayed_open in _initialize_s3_fs, controlled by a FileIO property such as s3.allow-delayed-open. I'd suggest defaulting it to true, since it only changes how small objects are uploaded. pyiceberg supports pyarrow 18 and up, so the option would be passed only on pyarrow 21 or later.

The workaround today is to subclass PyArrowFileIO, override _initialize_s3_fs, and rebuild the filesystem with the option added. That depends on a private method.

Willingness to contribute
  • I can contribute a fix for this bug independently
  • I would be willing to contribute a fix for this bug with guidance from the Iceberg community
  • I cannot contribute a fix for this bug at this time
Linguagem predominante
Python
Estrelas
1.2k
Forks
618
Merge médio
1d 13h
PRs com merge (30d)
47

Preparar o ambiente

  • Sem Dockerfile nem arquivo Docker Compose
  • Tem um modelo de pull request
  • Sem guia de contribuição

Primeiros passos

  1. Leia a issue inteira e depois o guia de contribuição do projeto.
  2. Comente na issue dizendo que vai assumir — evita que duas pessoas façam o mesmo trabalho.
  3. Faça um fork do repositório e trabalhe em uma branch.
  4. Abra um pull request que referencie o número da issue.

Mais de apache/iceberg-python

Todas as issues de apache/iceberg-python

Issues semelhantes

Mais issues de Python

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.