One of the two 4 TB drives in the backup box started logging pending sectors. The pool was still ONLINE, but a mirror with one doubtful member is a mirror in name only, so the drive went.
Device names move around between boots. Use the stable IDs and write the serial on the drive caddy before you open the case.
zpool status -P tank
ls -l /dev/disk/by-id/ | grep -v partOffline the old member, power down, replace, and let ZFS resilver onto the new disk:
zpool offline tank /dev/disk/by-id/ata-OLD_SERIAL
# swap the drive
zpool replace tank /dev/disk/by-id/ata-OLD_SERIAL /dev/disk/by-id/ata-NEW_SERIAL
zpool status tank # watch the resilverResilvering 2.1 TB of data took a little under five hours. The box stayed usable the whole time, just slower.
Run a scrub once the resilver finishes, and only then wipe the old drive. A monthly scrub from a systemd timer has caught two problems before they became interesting.
A mirror protects against a dead disk. It does not protect against rm -rf. Snapshots and an off-site copy do.