Corrupted filesystems on failed reconstruction

What happened:

On a busy machine, kubelet was restarted while a pod was deleted in the API server. Newly started kubelet tried to reconstruct the volume, but it failed with:

could not get consistent content of /proc/mounts after 3 attempts

Everything went wrong from here. Since kubelet can't reconstruct the volume, it only unmounts the pod local mount, leaving the global mount still mounted. And kubelet removed the volume from volumesInUse, therefore allowing the mounted volume to be detached + attached somewhere else. In case of FibreChannel or iSCSI, that don't have 3rd party attach, the volume ends up mounted on two different nodes and the volume gets corrupted. In clouds, the volume may get stuck attached to the node (because some clouds refuse to detach mounted volumes) or it gets force-detached, possibly corrupting the volume.

Exact sequence:

I0507 14:02:18.951820    4790 reconciler.go:411] "Could not construct volume information, cleaning up mounts" podName=8ca6b5d0-40f8-4232-a7f3-38e1000252e6 volumeSpecName="pv1" error="could not get consistent content of /proc/mounts after 3 attempts"
I0507 14:02:18.951847    4790 reconciler.go:444] "Reconciler sync states: could not find volume information in desired state, clean up the mount points" podName=8ca6b5d0-40f8-4232-a7f3-38e1000252e6 volumeSpecName="pv1"
...
I0507 14:02:18.952092    4790 subpath_linux.go:226] Cleaning up subpath mounts for /var/lib/kubelet/pods/8ca6b5d0-40f8-4232-a7f3-38e1000252e6/volume-subpaths/pv1
I0507 14:02:18.952170    4790 mount_helper_common.go:99] "/var/lib/kubelet/pods/8ca6b5d0-40f8-4232-a7f3-38e1000252e6/volumes/kubernetes.io~fc/pv1" is a mountpoint, unmounting
I0507 14:02:18.952180    4790 mount_linux.go:266] Unmounting /var/lib/kubelet/pods/8ca6b5d0-40f8-4232-a7f3-38e1000252e6/volumes/kubernetes.io~fc/pv1
W0507 14:02:18.960657    4790 mount_helper_common.go:129] Warning: "/var/lib/kubelet/pods/8ca6b5d0-40f8-4232-a7f3-38e1000252e6/volumes/kubernetes.io~fc/pv1" is not a mountpoint, deleting
I0507 14:02:18.960745    4790 operation_generator.go:829] UnmountVolume.TearDown succeeded for volume "pv1" (OuterVolumeSpecName: "") pod "8ca6b5d0-40f8-4232-a7f3-38e1000252e6" (UID: "8ca6b5d0-40f8-4232-a7f3-38e1000252e6"). InnerVolumeSpecName "pv1". PluginName "kubernetes.io/fc", VolumeGidValue ""
E0507 14:02:18.960775    4790 operation_generator.go:844] UnmountVolume.MarkVolumeAsUnmounted failed for volume "" (UniqueName: "pv1") pod "8ca6b5d0-40f8-4232-a7f3-38e1000252e6" (UID: "8ca6b5d0-40f8-4232-a7f3-38e1000252e6") : no volume with the name "pv1" exists in the list of attached volumes

+ the next node sync removed the volume from volumesInUse.

What you expected to happen:

The volume is not corrupted. Ideally, the volume is unmounted from global mount using UnmountDevice, however, it's not possible without reconstructed volume.

How to reproduce it (as minimally and precisely as possible):

I injected error to ConstructVolumeSpec of FibreChannel. I don't know how to reproduce it without it - it's hard to get GetMountRefs / ConsistentRead to return an error.

Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version):
• Cloud provider or hardware configuration:
• OS (e.g: cat /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Network plugin and version (if this is a network-related bug):
• Others:

/sig storage /priority important-soon