Kubernetes client exception with connection refused error

Our tests had been running fine till we started running into the following issue where the kubernetes client websocket call will terminate with an exception:

channel = stream(api.connect_get_namespaced_pod_exec,

/usr/lib/python3.8/site-packages/kubernetes/stream/stream.py:35: in _websocket_request return api_method(*args, **kwargs) /usr/lib/python3.8/site-packages/kubernetes/client/api/core_v1_api.py:994: in connect_get_namespaced_pod_exec return self.connect_get_namespaced_pod_exec_with_http_info(name, namespace, **kwargs) # noqa: E501 /usr/lib/python3.8/site-packages/kubernetes/client/api/core_v1_api.py:1101: in connect_get_namespaced_pod_exec_with_http_info return self.api_client.call_api( /usr/lib/python3.8/site-packages/kubernetes/client/api_client.py:348: in call_api return self.__call_api(resource_path, method, /usr/lib/python3.8/site-packages/kubernetes/client/api_client.py:180: in __call_api response_data = self.request(

---

configuration = <kubernetes.client.configuration.Configuration object at 0x7f192844cb20> _method = 'GET' url = 'wss://api.ndal-ocp2.ve.pd.f5net.com:6443/api/v1/namespaces/spk-test-automation/pods/ans-client-ci6492742-127/exec?com...command=2001%3A0db8%3A0%3A127%3A%3A100&command=-t&command=10&command=-P8&stderr=True&stdin=False&stdout=True&tty=False' kwargs = {'_preload_content': False, '_request_timeout': None, 'body': None, 'headers': {'Accept': '/', 'User-Agent': 'OpenAPI-Generator/25.3.0/python', 'authorization': 'Bearer sha256bgMewSf5E2EdW_h4EYLI4Ju-I21kPKVzK5tRTvqFgtg'}, ...} headers = {'Accept': '/', 'User-Agent': 'OpenAPI-Generator/25.3.0/python', 'authorization': 'Bearer sha256bgMewSf5E2EdW_h4EYLI4Ju-I21kPKVzK5tRTvqFgtg'} _request_timeout = None, _preload_content = False, capture_all = True def websocket_call(configuration, _method, url, **kwargs): """An internal function to be called in api-client when a websocket connection is required. method, url, and kwargs are the parameters of apiClient.request method."""

    url = get_websocket_url(url, kwargs.get("query_params"))
    headers = kwargs.get("headers")
    _request_timeout = kwargs.get("_request_timeout", 60)
    _preload_content = kwargs.get("_preload_content", True)
    capture_all = kwargs.get("capture_all", True)

    try:
        client = WSClient(configuration, url, headers, capture_all)
        if not _preload_content:
            return client
        client.run_forever(timeout=_request_timeout)
        return WSResponse('%s' % ''.join(client.read_all()))
    except (Exception, KeyboardInterrupt, SystemExit) as e:

      raise ApiException(status=0, reason=str(e))

E kubernetes.client.exceptions.ApiException: (0) E Reason: [Errno 111] Connection refused /usr/lib/python3.8/site-packages/kubernetes/stream/ws_client.py:525: ApiException

These exceptions happen at random (not for any specific or every test) and across different kubernetes environment/clusters. They started happening somewhere around early Feb, 2023. (not sure if its related to any fixes/changes).

Below is what we use in our code: `def k8s_exec(command, pod, namespace, api): channel = stream(api.connect_get_namespaced_pod_exec, pod, namespace, command=command, stderr=True, stdin=False, stdout=True, tty=False, _preload_content=False) channel.run_forever(timeout=60) err = yaml.safe_load(channel.read_channel(ERROR_CHANNEL)) if err is None: raise AssertionError(f'The command "{command}" was timeouted') rc = 0 if err['status'] == 'Success' else 1 stdout = channel.read_stdout() stderr = channel.read_stderr()

return rc, stdout, stderr`

Not sure if this is similar to: 1841

What you expected to happen: Connection is retained.

Environment:

• Kubernetes version (kubectl version): Client Version: version.Info{Major:"1", Minor:"20" Server Version: version.Info{Major:"1", Minor:"23"
• OS (e.g., MacOS 10.13.6): Linux
• Python version (python --version): 2.7.5