`put_async(Path)` opens files synchronously while holding the GIL

Aperta
  • Qualcosa non va?
#809 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
48/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
python, rust
Ambito
backend, performance

Direzione di ricerca

Start by tracing put_async() argument extraction for native Path inputs and the Rust async upload path; the issue names no source files or tests. Check how file opening and seek based size detection interact with cancellation and multipart selection. Done means slow opens do not block the event loop or hold the GIL, with the existing reader and upload behavior preserved.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Passing a pathlib.Path to obstore.put_async() opens the file during PyO3 argument extraction, before returning the awaitable. This happens synchronously while holding the Python GIL.

If opening the file stalls, it blocks the calling event loop. Moving the call to a Python worker thread does not fully address this: the file open still holds the GIL, preventing other Python threads from progressing.

Reproduction

On Unix, a named pipe provides a deterministic way to simulate a slow open. An external process supplies data after one second:

import asyncio
import os
import subprocess
import sys
import tempfile
from pathlib import Path

import obstore
from obstore.store import MemoryStore


async def main():
    with tempfile.TemporaryDirectory() as directory:
        source = Path(directory) / "source"
        os.mkfifo(source)

        writer = subprocess.Popen([
            sys.executable,
            "-c",
            """
import sys
import time

time.sleep(1)
with open(sys.argv[1], "wb") as output:
    output.write(b"hello")
""",
            str(source),
        ])

        async def heartbeat():
            while True:
                print("tick", flush=True)
                await asyncio.sleep(0.1)

        heartbeat_task = asyncio.create_task(heartbeat())
        try:
            await asyncio.sleep(0.1)
            await obstore.put_async(
                MemoryStore(),
                "destination",
                source,
                use_multipart=False,
            )
        finally:
            heartbeat_task.cancel()
            await asyncio.gather(heartbeat_task, return_exceptions=True)
            await asyncio.to_thread(writer.wait)


asyncio.run(main())

Observed: heartbeats stop while the file is opening.

Expected: local file preparation should not block the Python event loop or hold the GIL during blocking filesystem operations.

Possible fix

Defer opening native Path inputs until the Rust async operation runs, using tokio::task::spawn_blocking for opening and any seek-based size detection. Preserve the existing native reader and upload pipeline afterward.

A local prototype passes regression tests for event-loop responsiveness, cancellation during opening, and multipart selection. One observable behavior change is that file-opening errors would surface when awaiting the operation, rather than during the initial call.

If this fix is sound, I can open a PR with my local patch.

Reproduced with Obstore 0.11.1, CPython 3.14.7, macOS arm64.

Lingua principale
Python
Stelle
830
Fork
45
Merge medio
7h 53m
PR unite (30g)
13

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di developmentseed/obstore

Tutte le issue di developmentseed/obstore

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.