kubernetes-client/python

Fail watch gracefully on control plane upgrade

Open

#2462 opened on Oct 20, 2025

View on GitHub
 (6 comments) (0 reactions) (1 assignee)Python (6,225 stars) (3,323 forks)batch import
help wantedkind/bug

Description

What happened (please include outputs or screenshots): When I run a watch on k8s jobs (probably applies to other objects) and k8s control plane gets upgraded in the meantime, the watcher just silently stops producing events. No exception is being raised. What you expected to happen: Exception being raised or events continuing to arrive How to reproduce it (as minimally and precisely as possible):

  1. start watcher:
from kubernetes import client, config, watch

config.load_incluster_config()
batch_v1_api = client.BatchV1Api()
watcher = watch.Watch()

for event in watcher.stream(
        func=batch_v1_api.list_namespaced_job,
        namespace="mynamespace",
        timeout_seconds=0,
    ):
        print(event)
  1. upgrade k8s control plane
  2. create some new jobs
  3. new events are not coming in. no exceptions are being raised

Anything else we need to know?: Reproduced on GKE stable channel with the last two control plane upgrades Environment:

  • Kubernetes version (kubectl version): 1.33.5-gke.1080000
  • OS (e.g., MacOS 10.13.6): linux
  • Python version (python --version): 3.12.12
  • Python client version (pip list | grep kubernetes): 33.1.0

Contributor guide

Tech stack
python
Domain
api
Issue type
bug
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
over 1 week
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
clear
Prerequisites
Knowledge of Kubernetes watch APIPython client libraryAccess to a Kubernetes cluster with ability to upgrade
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
30
Research direction
Investigate the watch stream reconnection logic in the Python client. The issue likely arises when the control plane upgrade causes the watch connection to drop without proper error handling. Look at the `Watch` class in `kubernetes/watch/watch.py` and the underlying ` stream` method. Reproduce by simulating a connection reset in tests. Consider adding a heartbeat or timeout mechanism to detect stale watches.