CKS: nothing detects drift between a cluster's VM map, its VMs and its Kubernetes nodes
維護者通常 1 天內回覆
評估
- 難度
- 4/5
- 預估耗時
- 3-5 天
- 新手友好度
- 42/100
- Issue 類型
- 功能
- 描述清晰度
- 基本清楚
- 活躍度
- 活躍
- 技術堆疊
- java, kubernetes
- 領域
- cloud, observability
研究方向
Start by reading the CKS cluster status scanner and its handling of cluster VMs, then trace how it can access the cluster's Kubernetes nodes and raise alerts and events. Compare the VM map, actual VMs, and Kubernetes nodes for each mismatch. Done means each mismatch produces an alert and event naming the cluster and affected VM or node once, without changing the map.
由索引模型根據 Issue 內容生成。
描述
problem
CKS keeps the list of VMs of each Kubernetes cluster in kubernetes_cluster_vm_map. Scale, upgrade, stop/start and the cluster status scanner all trust that list. Nothing checks it against the VMs that actually exist or against the nodes Kubernetes knows about, so when the two drift apart nobody is told:
- a VM that is in the map but has been destroyed or expunged, or no longer exists, keeps the cluster in Alert (the scanner compares the map's size and VM states against the expected node count), and the next scale or upgrade works from a wrong list;
- a VM that runs as a node of the cluster but is not in the map is invisible to CloudStack: it is not scaled, upgraded, stopped or deleted with the cluster;
- a Kubernetes node with no VM behind it, or a mapped running VM that is not a Kubernetes node, means the cluster Kubernetes runs is not the one CloudStack thinks it manages.
The API and UI refuse to destroy a VM that belongs to a CKS cluster, so the map does not drift through normal use. It drifts when:
- a node is rebuilt or replaced by hand (a new VM joined to the cluster, the old one removed outside CKS);
- the database is edited by hand;
- a destroy or expunge fails during a scale-down, which removed the VM from the map but could leave the VM behind. That path is fixed in #14611.
A production cluster was found sitting in Alert with one of its control nodes missing from the map. The only way to find out why was to compare the map, the VMs and kubectl get nodes by hand.
versions
4.22 and main.
The steps to reproduce the bug
- Create a CKS cluster and wait for it to be Running.
- Remove one of its rows from
kubernetes_cluster_vm_map(or replace a worker node by hand with a new VM that joins the cluster). - The cluster moves to Alert, or keeps a node CloudStack does not know about, and no alert or event says which VM or node is the cause.
What to do about it?
Have the cluster status scanner compare the map with the cluster's VMs and its Kubernetes nodes, and raise an alert and an event naming the cluster and the VM or node for each mismatch, once per mismatch. Detection only, the map is not changed.
- 主要語言
- Java
- 星號
- 3.1k
- 分支
- 1.4k
- 平均合併
- 7 天 16 小時
- 30 天內合併 PR
- 15
環境準備
- 沒有 Dockerfile 或 Docker Compose 檔案
- 有 Pull Request 範本
- 閱讀貢獻指南
從這裡開始
- 先讀完整個 Issue，再讀專案的貢獻指南。
- 在 Issue 下留言說明你要接手 —— 這能避免兩個人做同樣的事。
- Fork 儲存庫，在一個分支上完成修改。
- 送出 Pull Request，並在描述裡引用這個 Issue 編號。
apache/cloudstack 的其他 Issue
-
resizeVolume of a running system VM's volume is accepted but the VM never sees the new size可能已有人在做 @bhouse-nexthop 於 4 天前認領。 未關閉
難度 2/5 1-3 小時 新手友好度 72/100
apache/cloudstack#14645 · 1 則留言 ·
維護者通常 1 天內回覆
-
VR reports a load balancer change as applied when haproxy.cfg cannot be replaced可能已有人在做 @bhouse-nexthop 於 4 天前認領。 未關閉
難度 2/5 1-3 小時 新手友好度 78/100
apache/cloudstack#14633 · 1 則留言 ·
維護者通常 1 天內回覆
-
restartNetwork reports a ResourceUnavailableException as an unexpected error (530), unlike restartVPC (534)可能已有人在做 @bhouse-nexthop 於 4 天前認領。 未關閉
難度 2/5 1-3 小時 新手友好度 78/100
apache/cloudstack#14631 · 1 則留言 ·
維護者通常 1 天內回覆
-
難度 2/5 1-3 小時 新手友好度 68/100
apache/cloudstack#14625 · 1 則留言 ·
維護者通常 1 天內回覆
-
Remote access VPN users are disconnected on every LB, ACL, port forward or firewall change (xl2tpd restarted on every iptables pass)可能已有人在做 @willbennett-devops 於 12 天前認領。 未關閉
難度 2/5 1-3 小時 新手友好度 72/100
apache/cloudstack#14545 · 1 則留言 ·
維護者通常 1 天內回覆
查看 apache/cloudstack 的全部 Issue
相似的 Issue
-
[FEATURE] Expose external refresh, node-level translog and warmer stats可能已有人在做 關聯的 PR 仍在進行中或已合併。 未關閉untriaged
難度 2/5 1-3 小時 新手友好度 68/100
opensearch-project/opensearch-prometheus-exporter#680 ·
維護者通常 2 天內回覆
-
難度 2/5 1-3 小時 新手友好度 85/100
-
難度 2/5 1-3 小時 新手友好度 66/100
維護者通常 2 天內回覆
-
bug
難度 1/5 1 小時以內 新手友好度 78/100
-
Good First Issue
難度 1/5 1-3 小時 新手友好度 88/100
hiero-ledger/hiero-sdk-java#3059 ·
維護者通常 1 天內回覆