Streaming jobs can bypass timeouts when output has no newline

The issue is in packages/syft-job/src/syft_job/job_runner.py in the _execute_job_streaming() function. Start by understanding the selector loop and the readline() call. Look at how timeouts are currently checked. The fix likely involves reading available bytes without blocking, perhaps using os.read or checking fileobj.read(1) in a non-blocking mode. Write a test that reproduces the bug using a script that outputs without a newline and sleeps. Verify the timeout is enforced after your changes.

Description

The streaming job runner can block while reading output that does not end with a newline, preventing the configured timeout from being enforced.

In packages/syft-job/src/syft_job/job_runner.py, _execute_job_streaming() waits for readable output using a selector and then calls:

line = key.fileobj.readline()

A pipe becoming readable only guarantees that some data is available. It does not guarantee that a complete line is available. If a job writes partial output without a newline and then continues running, readline() can block until the process writes a newline or exits.

While blocked inside readline(), the loop cannot execute its timeout check. If the process exits before readline() returns, the job may finish without being marked as timed out even though it exceeded the configured timeout.

How to Reproduce

1. Create a job whose run.sh writes output without a trailing newline and then sleeps:

   #!/bin/bash
   printf "working"
   sleep 10
   

2. Run the job using streaming output with a timeout shorter than the sleep duration, for example:

   timeout=1
   

3. Observe that the selector detects the available working output.

4. The subsequent readline() call waits for the process to exit because no newline was written.

5. The one-second timeout is not enforced while the runner is blocked.

A minimal Python reproduction of the underlying behavior is:

import selectors
import subprocess
import time

process = subprocess.Popen(
    ["bash", "-c", "printf working; sleep 5"],
    stdout=subprocess.PIPE,
    text=True,
)

selector = selectors.DefaultSelector()
selector.register(process.stdout, selectors.EVENT_READ)

start = time.monotonic()

for key, _ in selector.select(timeout=0.1):
    output = key.fileobj.readline()
    print(output, time.monotonic() - start)

Although output becomes available immediately, readline() returns only after approximately five seconds when the process exits.

Expected Behavior

The streaming runner should enforce the configured timeout regardless of whether job output contains newline characters.

Reading available output should not block the monitoring loop. Possible approaches include non-blocking reads, reading available chunks instead of complete lines, or otherwise ensuring that timeout checks continue while partial output is buffered.

Tests should cover a process that writes partial output without a newline and then runs longer than its configured timeout.

Screenshots

Not applicable. This behavior occurs in the job runner’s process-monitoring logic.

System Information

• OS: Reproduced on a POSIX environment
• OS Version: Not applicable
• Language Version: Python
• Package Manager Version: Not applicable
• Browser: Not applicable
• Browser Version: Not applicable

Additional Context

The non-streaming execution path uses process.communicate(timeout=timeout) and is not affected by this specific blocking readline() behavior.

Duplicate searches for streaming output without newlines, selector/readline timeout behavior, and hanging job timeouts found no matching open or closed issue.