bug(trainer): wait_for_job_status() accepts zero or negative polling_interval causing CPU busy-loop or cryptic error

What happened?

TrainerClient.wait_for_job_status() accepts a polling_interval parameter but performs no validation on its value. When a user passes 0 or a negative number, one of two silent failure modes occurs:

• polling_interval=0 → time.sleep(0) is called in the while True: polling loop, causing a CPU busy-loop that hammers the Kubernetes API thousands of times per second with no visible error.

• polling_interval=-5 → Python raises a cryptic stdlib error (ValueError: sleep length must be non-negative) deep inside the call stack with no hint that the caller passed a bad value.

Neither case gives the user a helpful, actionable error message.

Method signature (from kubeflow/trainer/api/trainer_client.py):

def wait_for_job_status(

    self,

    name: str,

    status: set[str] = {"Complete"},

    timeout: int = 600,

    polling_interval: int = 2,   # ← no validation

    callbacks: list[Callable[[TrainJob], None]] | None = None,

) -> TrainJob:

Steps to reproduce:

from kubeflow.trainer import TrainerClient



client = TrainerClient()

# Triggers CPU busy-loop:

client.wait_for_job_status("my-job", polling_interval=0)



# Triggers cryptic stdlib ValueError:

client.wait_for_job_status("my-job", polling_interval=-5)

Note: The same gap exists in ContainerBackend.wait_for_job_status() (tracked in #411). This issue covers the Kubernetes backend path in TrainerClient.

What did you expect to happen?

A clear, descriptive ValueError should be raised immediately when polling_interval <= 0, before any polling loop begins. Example:

ValueError: polling_interval must be a positive number, got: 0

This is consistent with how the SDK already documents ValueError in the wait_for_job_status() raises section ("The input values are incorrect") — the validation just needs to be implemented.

Environment

• Kubeflow SDK version: v0.4.0 (latest)
• Affected file: kubeflow/trainer/api/trainer_client.py
• Affected method: TrainerClient.wait_for_job_status()
• Related issue: #411 (same bug in ContainerBackend)
• Python: 3.9+
• Backend: KubernetesBackend (default)

Impacted by this bug?

Proposed Fix Add input validation at the top of wait_for_job_status() in kubeflow/trainer/api/trainer_client.py: python if not isinstance(polling_interval, (int, float)) or polling_interval <= 0: raise ValueError( f"polling_interval must be a positive number, got: {polling_interval}" ) Files to change: 1. kubeflow/trainer/api/trainer_client.py — add the guard clause above 2. test/unit/trainer/test_trainer_client.py — add unit tests for polling_interval=0 and polling_interval=-5 Test cases to add: python def test_wait_for_job_status_invalid_polling_interval(): client = TrainerClient() with pytest.raises(ValueError, match="polling_interval"): client.wait_for_job_status("some-job", polling_interval=0) with pytest.raises(ValueError, match="polling_interval"): client.wait_for_job_status("some-job", polling_interval=-5) --- I would like to work on this issue. Could a maintainer please assign it to me? 🙏 My plan: 1. Add the ValueError guard in trainer_client.py 2. Add unit tests covering both failure modes 3. Update the docstring Raises section to explicitly mention polling_interval 4. Open a PR referencing this issue and #411 for consistency