Feature Request: replace a single Machine with surge (create-before-delete)

Abierto
  • ¿Algo no está bien?
#2,104 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
12/100
Tipo de issue
Nueva funcionalidad
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
go, kubernetes
Área
infrastructure

Línea de trabajo

Start with shouldExcludeMachine in pkg/controller/machineset/controller.go, which decides why a Machine with a DeletionTimestamp stops counting toward replicas. Then read the surge and quota handling the issue links to in #1968. The issue asks for a design, not a patch: done means the annotation, the surge-and-swap sequence, failure handling and the rollout interaction are agreed with maintainers first.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

customer-request kind/feature

Problem

Deleting a single Machine in a MachineDeployment causes a temporary capacity drop. The MachineSet creates a replacement immediately, but the old node is drained and removed in parallel, which usually finishes before the new node has joined.

Rolling the whole MachineDeployment avoids this, because maxSurge/maxUnavailable only remove old Machines once new ones are available. There is no equivalent for replacing one specific Machine. Typical cases:

  • a degraded or misbehaving node
  • a node that needs a fresh OS image
  • a node on a host scheduled for maintenance

Root cause

Machines with a DeletionTimestamp are excluded from the MachineSet's replica count (shouldExcludeMachine in pkg/controller/machineset/controller.go). Replacement and deletion therefore run concurrently, with no availability guarantee.

Proposal

Add a Machine annotation, for example machine-controller.kubermatic.io/replace=true, handled by the MachineSet controller:

  1. Keep counting the annotated Machine as live and create one additional Machine.
  2. Once the new Machine's node is Ready and has passed minReadySeconds, delete the annotated Machine. The existing drain path handles eviction and respects PDBs.
  3. Report progress via events and a condition on the Machine (for example
    ReplacementPending, ReplacementReady, ReplacementFailed).

Design considerations

  • Quota: the surge Machine is subject to project quota validation, the same as Machines created during rolling updates (see #1968). A replace keeps net replicas unchanged, so it should be eligible for whatever quota handling #1968 settles on,
    rather than needing a separate mechanism.
  • Surge failure: quota denial, IP-pool exhaustion or provider errors need a timeout or abort path that removes the stuck Machine and leaves the original running. A failed replace must never reduce capacity.
  • Interaction with an in-progress rollout: either block replacements while a rollout is active, or count the surge against the MachineDeployment's maxSurge.
  • Concurrency: cap how many annotated Machines are replaced at once.
  • Cluster autoscaler: prevent the surge node from being scaled down before the swap completes.

Current workaround

  1. Annotate the target Machine with cluster.k8s.io/delete-machine=yes and cordon its node.
  2. Scale the MachineDeployment to N+1 and wait for the new node to be Ready.
  3. Scale back to N. The annotated Machine is selected for deletion.

Limitations of this workaround:

  • Manual and racy: a concurrent replicas change or the autoscaler can interfere.
  • Wrong Machine possible: under the default Random delete policy, an errored Machine has the same delete priority as the annotated one and may be picked instead.
  • Blocked by quota: step 2 is a genuine scale-up, so it is denied in projects near their quota limit. It would not qualify for a rolling-update quota bypass like the one proposed in #1968.

Related

  • #1968: surge Machines created during rolling updates are denied by project quota validation. A native single-Machine replace has the same create-before-delete
    shape and would hit the same denial in projects near their quota limit.

Additional Information

Internal Reference: 10357

Lenguaje dominante
Go
Estrellas
325
Forks
140
Merge medio
4 h 3 min
PR fusionados (30 d)
13

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de kubermatic/machine-controller

Todos los issues de kubermatic/machine-controller

Issues similares

Más issues de Go

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.