OpenZFS, it turns out, can react very poorly to a reshuffling of the disks. In one zpool I had six disks, arranged as 3 mirrored VDEVs, physically arranged in two external enclosures. One of the enclosure has 4 slots, the other 2. Two of the VDEVs were in one enclosure, the third in the second. This isn’t ideal: one enclosure failure will offline the whole zpool. To fix this, I got ahold of another 4 slot enclosure to spread the VDEVs around. That’s when things went a little fucky.
To improve resilience I did a reshuffle, moving half of each VDEV into the new 4 slot enclosure, and half of each VDEV into the old 4 slot enclosure. As well as freeing up another pair of slots, this ensure that an enclosure failure won’t offline the zpool, just half of each mirror. I decided to do this re-org cold, powering down the server the storage is attached to.
When it rebooted, the ZFS scripts aut-imported the pool and… all the mirrors were in a degraded state, with half the devices reporting UNAVAIL and reporting their ZFS GUID and Formerly /dev/disk/by-id/xxxxx in the right-hand column.
Fuck.
Fixing this ended up going up a lot of blind alleys, in part because trying to zfs offline and zfs replace the drives (as suggested by Dr. Google) didn’t work, complaining instead that the failed disks were part of an active array. The root cause of the confusion appeared to be that the old disk-by-id mapping was now occupied by a different disk. At the point at which Dr. Google suggested scrubbing the label off the failed disk I decided there had to be a better (and less potentially destructive) way and tried:
zfs offline archives 16830548349776597394
zfs export archives
zfs import archives
zfs online archives 16830548349776597394(In the above example, the number is the ZFS UUID, which can always be used for zpool ops, even when zfs is bitterly confused about paths and such.)
At this point, the disks are actived under their new path and begin to re-silver.