projectcalico/calico

Update Kubernetes Node condition `NetworkUnavailable` in case of problems

Open

#5,233 opened on Jun 16, 2020

View on GitHub
 (17 comments) (16 reactions) (0 assignees)Go (1,594 forks)auto 404
help wantedkind/enhancement

Repository metrics

Stars
 (7,292 stars)
PR merge metrics
 (PR metrics pending)

Description

Hi, we are sometimes encountering problems with calico-node on Kubernetes.

Issue Description The condition type NetworkUnavailable on a Kubernetes node seems to be only initialised to false on startup. However, sometimes we are experiencing issues such as (from calico-node logs):

bird: Mesh_10_250_10_60: Socket error: bind: Address in use
bird: Mesh_10_250_19_13: Socket error: bind: Address in use
bird: Mesh_10_250_21_169: Socket error: bind: Address in use

The aforementioned node condition is still set to False with reason CalicoIsUp indicating that everything is working fine.

Expected Behavior

I would expect that calico's state is reflected in the Kubernetes node condition NetworkUnavailable (also in case of abnormalities during runtime).

Current Behavior

The condition type NetworkUnavailable on a Kubernetes node is only initialised to false on startup so problem during runtime are not reflected in the node condition.

Another possibility is, that calico sets the NetworkUnavailable too early during startup, with the actual error happening after the initialisation.

Possible Solution

Steps to Reproduce (for bugs)

Context

We are creating Kubernetes node via the Machine Controller Manager with calcio-node as deamon set. We are monitoring the Node conditions including the NetworkUnavailable to be able to replace a node in case calico has problems.

Your Environment

  • calico/node version:
  • Operating System and version: Ubuntu 18.04.4 LTS (though observed independently)
  • Link to your project (optional):

Contributor guide