Concurrent resilvers of multiple drives are inconsistently concurrent and/or inconsistently report status
#9,846 opened on 2020年1月15日
説明
System information
| Type | Linux x86-64 CentOS 7.7 |
|---|---|
| Distribution Name | CentOS |
| Distribution Version | 7.7.1908 |
| Linux Kernel | 3.10.0-1062.9.1.el7.x86_64 |
| Architecture | x86_64 |
| ZFS Version | 0.8.2-1 |
| SPL Version | 0.8.2-1 |
Describe the problem you're observing
Over the recent vacations, I decided to replace three drives in our ZFS pool which had been showing occasional read errors. This should be a fairly fast procedure, because the pool would be pretty much completely idle. The pool consists of 6 raidz2 vdevs with 12 10TB SAS drives in each. Coincidentally, the three drives I wanted to replace were all in different vdevs.
I manually faulted all three drives with zpool offline -f <olddrive>, and then replaced each drive physically in the enclosure, and did a zpool replace <olddrive> <newdrive>. Time between the three replace commands was in the range of 5-8 minutes, basically the time it took to remove the drive, get the new one in the sled, and replace it.
After I had done this, zpool status showed the pool as resilvering, and each of the three devices had (resilvering) after them in the list of devices. After a while, the time estimate stabilized at about 60 hours total, which seemed fine.
I let this run until completion, logging in remotely about when it was scheduled to finish to check that everything was ok. To my surprise, when the resilver finished, it immediately started again, but now zpool status showed the first drive I'd replaced as ONLINE, while the other two were again marked as (resilvering). The time estimate again was about 60 hours after stabilizing. At this point, I prepared myself for this resilver to finish and then to start again with only the last drive, but surprisingly, after another two and a half days, the second resilver finished, and all drives show up as ONLINE.
There seems to be several suboptimal behaviors here. One is that, obviously ZFS is capable of concurrently resilvering several drives in a pool (at least when they're in different vdevs, as was the case here, I've never tried doing it in the same vdev, and probably wouldn't unless the drives were totally dead), but it sometimes chooses to do it and other times not.
The other problem is that, if ZFS is really just resilvering one drive out of several, it should probably say so somehow. There's no way I could find out to tell if one or several drives were being resilvered, they all showed up as (resilvering) in zpool status.