kubernetes-sigs/cluster-api

Deleting InfraMachine could start before detaching all volumes finishes completely

Open

#8,507 opened on Apr 11, 2023

View on GitHub
 (13 comments) (1 reaction) (0 assignees)Go (1,532 forks)auto 404
help wantedkind/bugpriority/backlogtriage/accepted

Repository metrics

Stars
 (4,267 stars)
PR merge metrics
 (PR metrics pending)

Description

What steps did you take and what happened?

  1. Using CAPI to create Kubernetes Cluster.
  2. Create a pod mounted 10 pvc.
  3. After waiting for the Pod to run, delete the Machine where the Pod is located.

Sometimes delete infra machine will happened before detaching all volume finished when deleting machine.

The relate log is

I0410 09:25:54.828555       1 machine_controller.go:383] "Waiting for node volumes to be detached" controller="machine" controllerGroup="cluster.x-k8s.io" controllerKind="Machine" Machine="default/rockey-elf-csi-test-workergroup-9d684b695x4gl9w-bfsvh" namespace="default" name="rockey-elf-csi-test-workergroup-9d684b695x4gl9w-bfsvh" reconcileID=59a8afde-a1ee-4302-a4f2-c7853c653c91 MachineSet="default/rockey-elf-csi-test-workergroup-9d684b695x4gl9w" MachineDeployment="default/rockey-elf-csi-test-workergroup" Cluster="default/rockey-elf-csi-test" Node="rockey-elf-csi-test-workergroup-sgsqt"
I0410 09:25:55.411311       1 machine_controller.go:383] "Waiting for node volumes to be detached" controller="machine" controllerGroup="cluster.x-k8s.io" controllerKind="Machine" Machine="default/rockey-elf-csi-test-workergroup-9d684b695x4gl9w-bfsvh" namespace="default" name="rockey-elf-csi-test-workergroup-9d684b695x4gl9w-bfsvh" reconcileID=1c9e8442-1383-4862-8426-ad915c0fc9a1 MachineSet="default/rockey-elf-csi-test-workergroup-9d684b695x4gl9w" MachineDeployment="default/rockey-elf-csi-test-workergroup" Cluster="default/rockey-elf-csi-test" Node="rockey-elf-csi-test-workergroup-sgsqt"
I0410 09:25:55.887963       1 machine_controller.go:383] "Waiting for node volumes to be detached" controller="machine" controllerGroup="cluster.x-k8s.io" controllerKind="Machine" Machine="default/rockey-elf-csi-test-workergroup-9d684b695x4gl9w-bfsvh" namespace="default" name="rockey-elf-csi-test-workergroup-9d684b695x4gl9w-bfsvh" reconcileID=d9e57f1c-a540-43d9-88f9-2102aa8ad832 MachineSet="default/rockey-elf-csi-test-workergroup-9d684b695x4gl9w" MachineDeployment="default/rockey-elf-csi-test-workergroup" Cluster="default/rockey-elf-csi-test" Node="rockey-elf-csi-test-workergroup-sgsqt"
I0410 09:25:56.337312       1 machine_controller.go:383] "Waiting for node volumes to be detached" controller="machine" controllerGroup="cluster.x-k8s.io" controllerKind="Machine" Machine="default/rockey-elf-csi-test-workergroup-9d684b695x4gl9w-bfsvh" namespace="default" name="rockey-elf-csi-test-workergroup-9d684b695x4gl9w-bfsvh" reconcileID=f62fd36c-f160-4735-b28f-108f48b9ba82 MachineSet="default/rockey-elf-csi-test-workergroup-9d684b695x4gl9w" MachineDeployment="default/rockey-elf-csi-test-workergroup" Cluster="default/rockey-elf-csi-test" Node="rockey-elf-csi-test-workergroup-sgsqt"
I0410 09:25:59.998949       1 machine_controller.go:417] "Waiting for infrastructure to be deleted" controller="machine" controllerGroup="cluster.x-k8s.io" controllerKind="Machine" Machine="default/rockey-elf-csi-test-workergroup-9d684b695x4gl9w-bfsvh" namespace="default" name="rockey-elf-csi-test-workergroup-9d684b695x4gl9w-bfsvh" reconcileID=d28b45ed-8497-4e19-8f4c-6bedc94773c5 MachineSet="default/rockey-elf-csi-test-workergroup-9d684b695x4gl9w" MachineDeployment="default/rockey-elf-csi-test-workergroup" Cluster="default/rockey-elf-csi-test" ElfMachine="default/rockey-elf-csi-test-workergroup-sgsqt"
I0410 09:26:00.496313       1 machine_controller.go:383] "Waiting for node volumes to be detached" controller="machine" controllerGroup="cluster.x-k8s.io" controllerKind="Machine" Machine="default/rockey-elf-csi-test-workergroup-9d684b695x4gl9w-bfsvh" namespace="default" name="rockey-elf-csi-test-workergroup-9d684b695x4gl9w-bfsvh" reconcileID=a5fc9a53-34dc-45d0-b340-4ddb3d07a7bc MachineSet="default/rockey-elf-csi-test-workergroup-9d684b695x4gl9w" MachineDeployment="default/rockey-elf-csi-test-workergroup" Cluster="default/rockey-elf-csi-test" Node="rockey-elf-csi-test-workergroup-sgsqt"
I0410 09:26:00.968387       1 machine_controller.go:383] "Waiting for node volumes to be detached" controller="machine" controllerGroup="cluster.x-k8s.io" controllerKind="Machine" Machine="default/rockey-elf-csi-test-workergroup-9d684b695x4gl9w-bfsvh" namespace="default" name="rockey-elf-csi-test-workergroup-9d684b695x4gl9w-bfsvh" reconcileID=7ee32cbd-6833-4da7-be50-9191ceae6187 MachineSet="default/rockey-elf-csi-test-workergroup-9d684b695x4gl9w" MachineDeployment="default/rockey-elf-csi-test-workergroup" Cluster="default/rockey-elf-csi-test" Node="rockey-elf-csi-test-workergroup-sgsqt"
I0410 09:26:01.417174       1 machine_controller.go:383] "Waiting for node volumes to be detached" controller="machine" controllerGroup="cluster.x-k8s.io" controllerKind="Machine" Machine="default/rockey-elf-csi-test-workergroup-9d684b695x4gl9w-bfsvh" namespace="default" name="rockey-elf-csi-test-workergroup-9d684b695x4gl9w-bfsvh" reconcileID=34a5cf22-2a5c-4ff5-a333-92f4aecb5826 MachineSet="default/rockey-elf-csi-test-workergroup-9d684b695x4gl9w" MachineDeployment="default/rockey-elf-csi-test-workergroup" Cluster="default/rockey-elf-csi-test" Node="rockey-elf-csi-test-workergroup-sgsqt"

The root case is CAPI using node.Status.VolumeAttached to check whether all volumes are detached when deleting machine, but in Kubernetes, volume record in node.Status.VolumeAttached will be removed before detached volume finished. https://github.com/kubernetes/kubernetes/blob/ad18954259eae3db51bac2274ed4ca7304b923c4/pkg/controller/volume/attachdetach/reconciler/reconciler.go#L239-L290

What did you expect to happen?

Delete infra machine should happened after detaching all volume finished when deleting machine.

Cluster API version

1.4.0

Kubernetes version

1.24.8

Anything else you would like to add?

https://kubernetes.slack.com/archives/C8TSNPY4T/p1681120800532549

Label(s) to be applied

/kind bug One or more /area label. See https://github.com/kubernetes-sigs/cluster-api/labels?q=area for the list of labels.

Contributor guide

Research direction
Examine the machine controller.go reconciliation logic to understand the sequence of volume detachment checks and infrastructure deletion. The issue suggests that the node volume status check may complete before actual detachment. Look at the reconciler logic, particularly the waiting loop and the transition to infrastructure deletion. Identify where a race condition exists and propose a fix to ensure volumes are fully detached before proceeding.
Tech stack
gokubernetes
Domain
backendinfrastructure
Issue type
Bug
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
2
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-3 hours
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Clear
Prerequisites
GoKubernetes
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
60