Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

Streaming jobs can bypass timeouts when output has no newline

Aperta
  • Qualcosa non va?
#9,528 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
3/5
Tempo stimato
1-2 giorni
Idoneità per principianti
65/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
python
Ambito
backend, cli, tooling

Direzione di ricerca

The issue is in packages/syft-job/src/syft_job/job_runner.py in the _execute_job_streaming() function. Start by understanding the selector loop and the readline() call. Look at how timeouts are currently checked. The fix likely involves reading available bytes without blocking, perhaps using os.read or checking fileobj.read(1) in a non-blocking mode. Write a test that reproduces the bug using a script that outputs without a newline and sleeps. Verify the timeout is enforced after your changes.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Type: Bug :bug:

Description

The streaming job runner can block while reading output that does not end with a newline, preventing the configured timeout from being enforced.

In packages/syft-job/src/syft_job/job_runner.py, _execute_job_streaming() waits for readable output using a selector and then calls:

line = key.fileobj.readline()

A pipe becoming readable only guarantees that some data is available. It does not guarantee that a complete line is available. If a job writes partial output without a newline and then continues running, readline() can block until the process writes a newline or exits.

While blocked inside readline(), the loop cannot execute its timeout check. If the process exits before readline() returns, the job may finish without being marked as timed out even though it exceeded the configured timeout.

How to Reproduce

  1. Create a job whose run.sh writes output without a trailing newline and then sleeps:

    #!/bin/bash
printf "working"
sleep 10

  2. Run the job using streaming output with a timeout shorter than the sleep duration, for example:

    timeout=1

  3. Observe that the selector detects the available working output.

  4. The subsequent readline() call waits for the process to exit because no newline was written.

  5. The one-second timeout is not enforced while the runner is blocked.

A minimal Python reproduction of the underlying behavior is:

import selectors
import subprocess
import time

process = subprocess.Popen(
    ["bash", "-c", "printf working; sleep 5"],
    stdout=subprocess.PIPE,
    text=True,
)

selector = selectors.DefaultSelector()
selector.register(process.stdout, selectors.EVENT_READ)

start = time.monotonic()

for key, _ in selector.select(timeout=0.1):
    output = key.fileobj.readline()
    print(output, time.monotonic() - start)

Although output becomes available immediately, readline() returns only after approximately five seconds when the process exits.

Expected Behavior

The streaming runner should enforce the configured timeout regardless of whether job output contains newline characters.

Reading available output should not block the monitoring loop. Possible approaches include non-blocking reads, reading available chunks instead of complete lines, or otherwise ensuring that timeout checks continue while partial output is buffered.

Tests should cover a process that writes partial output without a newline and then runs longer than its configured timeout.

Screenshots

Not applicable. This behavior occurs in the job runner’s process-monitoring logic.

System Information

  • OS: Reproduced on a POSIX environment
  • OS Version: Not applicable
  • Language Version: Python
  • Package Manager Version: Not applicable
  • Browser: Not applicable
  • Browser Version: Not applicable

Additional Context

The non-streaming execution path uses process.communicate(timeout=timeout) and is not affected by this specific blocking readline() behavior.

Duplicate searches for streaming output without newlines, selector/readline timeout behavior, and hanging job timeouts found no matching open or closed issue.

Lingua principale
Python
Stelle
10k
Fork
2k
Merge medio
23h 19m
PR unite (30g)
21

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di OpenMined/PySyft

Tutte le issue di OpenMined/PySyft

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.