Feature Request: replace a single Machine with surge (create-before-delete)

Đang mở
  • Có gì không ổn?
#2,104 0 bình luận 0 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

Chưa có ai nhận issue này.

Đánh giá

Độ khó
5/5
Thời gian dự kiến
Hơn một tuần
Mức phù hợp với người mới
12/100
Loại issue
Tính năng
Độ rõ ràng
Khá rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
go, kubernetes
Lĩnh vực
infrastructure

Hướng nghiên cứu

Start with shouldExcludeMachine in pkg/controller/machineset/controller.go, which decides why a Machine with a DeletionTimestamp stops counting toward replicas. Then read the surge and quota handling the issue links to in #1968. The issue asks for a design, not a patch: done means the annotation, the surge-and-swap sequence, failure handling and the rollout interaction are agreed with maintainers first.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

customer-request kind/feature

Problem

Deleting a single Machine in a MachineDeployment causes a temporary capacity drop. The MachineSet creates a replacement immediately, but the old node is drained and removed in parallel, which usually finishes before the new node has joined.

Rolling the whole MachineDeployment avoids this, because maxSurge/maxUnavailable only remove old Machines once new ones are available. There is no equivalent for replacing one specific Machine. Typical cases:

  • a degraded or misbehaving node
  • a node that needs a fresh OS image
  • a node on a host scheduled for maintenance

Root cause

Machines with a DeletionTimestamp are excluded from the MachineSet's replica count (shouldExcludeMachine in pkg/controller/machineset/controller.go). Replacement and deletion therefore run concurrently, with no availability guarantee.

Proposal

Add a Machine annotation, for example machine-controller.kubermatic.io/replace=true, handled by the MachineSet controller:

  1. Keep counting the annotated Machine as live and create one additional Machine.
  2. Once the new Machine's node is Ready and has passed minReadySeconds, delete the annotated Machine. The existing drain path handles eviction and respects PDBs.
  3. Report progress via events and a condition on the Machine (for example
    ReplacementPending, ReplacementReady, ReplacementFailed).

Design considerations

  • Quota: the surge Machine is subject to project quota validation, the same as Machines created during rolling updates (see #1968). A replace keeps net replicas unchanged, so it should be eligible for whatever quota handling #1968 settles on,
    rather than needing a separate mechanism.
  • Surge failure: quota denial, IP-pool exhaustion or provider errors need a timeout or abort path that removes the stuck Machine and leaves the original running. A failed replace must never reduce capacity.
  • Interaction with an in-progress rollout: either block replacements while a rollout is active, or count the surge against the MachineDeployment's maxSurge.
  • Concurrency: cap how many annotated Machines are replaced at once.
  • Cluster autoscaler: prevent the surge node from being scaled down before the swap completes.

Current workaround

  1. Annotate the target Machine with cluster.k8s.io/delete-machine=yes and cordon its node.
  2. Scale the MachineDeployment to N+1 and wait for the new node to be Ready.
  3. Scale back to N. The annotated Machine is selected for deletion.

Limitations of this workaround:

  • Manual and racy: a concurrent replicas change or the autoscaler can interfere.
  • Wrong Machine possible: under the default Random delete policy, an errored Machine has the same delete priority as the annotated one and may be picked instead.
  • Blocked by quota: step 2 is a genuine scale-up, so it is denied in projects near their quota limit. It would not qualify for a rolling-update quota bypass like the one proposed in #1968.

Related

  • #1968: surge Machines created during rolling updates are denied by project quota validation. A native single-Machine replace has the same create-before-delete
    shape and would hit the same denial in projects near their quota limit.

Additional Information

Internal Reference: 10357

Ngôn ngữ chính
Go
Star
325
Fork
140
Merge trung bình
1 giờ 35 phút
Pull request đã merge (30 ngày)
14

Chuẩn bị môi trường

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của kubermatic/machine-controller

Tất cả issue của kubermatic/machine-controller

Issue tương tự

Thêm issue về Go

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.