Replacing a failing disk in a mirrored vdev results in checksum errors on new drive

[aaronjwood]

This may be from my own misunderstanding: when replacing a disk in a mirrored vdev setup does ZFS read data from both of the disks at the same time when building out the new disk? My scenario played out like this:

1. One of the two disks in the mirrored vdev started to fail with uncorrectable sectors (1-2 per day)
2. I went to replace this disk but did not offline the old disk first
3. The replacement drive was a known working drive that I had used over the past year, just not in recent months as it became a spare for me
4. The new disk successfully resilvered and I removed the old disk from my JBOD
5. Every day I would see a handful of checksum errors on the new drive which were always repaired
6. My scheduled scrub eventually kicked in and encountered (but corrected) ~500 checksum errors
7. Multiple weeks went by after the scrub with zero checksum errors being reported

It looks like I should have explicitly offlined the old disk first before starting a replacement/resilver. Before I started the replacement process I had done a zpool clear, I'm not sure if this would change the behavior of ZFS or not. If ZFS knew the existing disk had errors and the same scenario above played out would ZFS have read data only from the one healthy disk in the mirrored vdev?

[yhay81]

You generally should not offline the old member first when it is still readable. Keeping it online preserves an additional copy during replacement. The documented semantics of zpool replace old new are effectively “attach the new device, resilver it, then detach the old device” (zpool-replace(8)).

ZFS does not need to choose one source disk for the entire resilver. Mirror reads can be distributed among eligible children; if a returned block fails its checksum or an I/O fails, redundancy lets ZFS read a valid replica and repair the bad copy. Therefore the important guarantee is checksum validation and retry, not whether both old members were being read simultaneously. Offlining the suspect member would merely remove one possible source and reduce the temporary redundancy.

The exception worth checking is whether you explicitly used zpool replace -s. The -s sequential-reconstruction mode does not verify checksums during reconstruction and deliberately starts a scrub afterward. A normal replacement/resilver is not that mode.

zpool clear did not cause ZFS to trust corrupt blocks or change the source-selection algorithm. It clears the recorded device error counters; it does not repair data or erase block checksums (zpool-clear(8)). OpenZFS's troubleshooting guide is explicit that clearing resets counters only, so the hardware cause should be fixed before clearing and then monitored for recurrence (troubleshooting).

The part I would investigate is the attribution to the new device. A CKSUM count on that vdev means a later read from that path returned bytes that did not match the block checksum; ZFS repaired them from another replica. It does not by itself mean “bad data was copied from the failing disk.” Repeated CKSUM with few/no READ or WRITE errors often points to the data path—SATA/SAS cable, backplane slot, HBA, power, or RAM—as well as the disk itself. The fact that a scrub repaired the blocks and later scrubs/read traffic have stayed clean is reassuring, but ~500 corrected blocks is enough to inspect rather than dismiss.

I would preserve the current evidence (and confirm the backup) before another zpool clear, then collect:

zpool status -vP <pool>
zpool events -v <pool>
smartctl -x /dev/disk/by-id/<replacement>
dmesg -T | grep -Ei 'ata|sas|scsi|reset|crc|i/o error|zfs'

For SATA, pay particular attention to UDMA CRC errors (cable/backplane) versus pending/offline-uncorrectable sectors (media). Run the replacement disk's long SMART self-test and keep monitoring across the next scrub. If CKSUM increments again, move/replace the cable or slot and treat the replacement disk as suspect until the source is isolated.

So the short answers are: leaving the old disk online was the correct replacement procedure; zpool clear only hid the previous counters; and the corrected errors on the new vdev warrant checking the new disk's entire I/O path rather than assuming ZFS resilvered from the wrong mirror member.