Zfs resilvering stuck in a loop.

[cntimis]

Hello,
I have a Centos 7 box

Kernel 5.2.9-1.el7.elrepo.x86_64
libzfs2-0.8.1-1.el7.x86_64
zfs-0.8.1-1.el7.x86_64
kmod-zfs-5.2.9-1.el7.elrepo.x86_64-0.8.1-1.el7.x86_64

with 4 raidz2 vdevs each on a separate enclosure. Two disks enc-dc66-phy7 enc-dc66-phy10 were in SMART predictive failure and I had this resilvering loop starting and stopping immediately afterwards. I've replaced one of them, enc-dc66-phy7, but it didn't start relivering and I still got this resilvering loop started as soon as I typed zpool replace ... command. Zpool status showed 'resilver in progress since ...' and the time and date of the moment when I typed the command and nothing resilvered on the actual replaced disk. Somehow after typing these commands, the resilvering of enc-dd66-phy7 has started and it took about 18 hours to complete:

 1311  2025-09-12 14:18:09 zpool status
 1312  2025-09-12 14:19:08 zpool offline  store1 enc-dc66-phy7
 1313  2025-09-12 14:19:18 zpool status
 1314  2025-09-12 14:20:02 less /var/log/messages
 1315  2025-09-12 14:22:37 zpool scrub -s store1
 1316  2025-09-12 14:22:46 zpool status
 1317  2025-09-12 14:23:15 zpool clear store1
 1318  2025-09-12 14:23:23 zpool status
 1319  2025-09-12 14:23:40 zpool online store1 enc-dc66-phy7
 1320  2025-09-12 14:23:53 zpool status

Now the second enc-dc66-phy10 disks of the two is replaced but again, as soon as I typed zpool replace ... the resilvering in a loop has started.

  pool: store1
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Mon Sep 15 15:37:28 2025
        202G scanned at 33.7G/s, 98.6G issued at 16.4G/s, 197T total
        0B resilvered, 0.05% done, 0 days 03:24:18 to go
config:

        NAME                  STATE     READ WRITE CKSUM
        store1             DEGRADED     0     0     0
          raidz2-0            DEGRADED     0     0     0
            enc-dc66-phy0     ONLINE       0     0     0
            enc-dc66-phy1     ONLINE       0     0     0
            enc-dc66-phy2     ONLINE       0     0     0
            enc-dc66-phy3     ONLINE       0     0     0
            enc-dc66-phy4     ONLINE       0     0     0
            enc-dc66-phy5     ONLINE       0     0     0
            enc-dc66-phy6     ONLINE       0     0     0
            enc-dc66-phy7     ONLINE       0     0     0
            enc-dc66-phy8     ONLINE       0     0     0
            enc-dc66-phy9     ONLINE       0     0     0
            replacing-10      DEGRADED     0     0     0
              old             OFFLINE      0     0     0
              enc-dc66-phy10  ONLINE       0     0     0
            enc-dc66-phy11    ONLINE       0     0     0
          raidz2-1            ONLINE       0     0     0
            enc-dd09-phy0     ONLINE       0     0     0
            enc-dd09-phy1     ONLINE       0     0     0

zpool events -v shows:

Sep 15 2025 16:18:24.014451199 sysevent.fs.zfs.history_event
        version = 0x0
        class = "sysevent.fs.zfs.history_event"
        pool = "store1"
        pool_guid = 0x4ab56a61b1481ffb
        pool_state = 0x0
        pool_context = 0x0
        history_hostname = "myhost"
        history_internal_str = "errors=0"
        history_internal_name = "scan aborted, restarting"
        history_txg = 0x2b0c09f
        history_time = 0x68c82e40
        time = 0x68c82e40 0xdc81ff
        eid = 0x328ec3

Sep 15 2025 16:18:24.021451334 sysevent.fs.zfs.history_event
        version = 0x0
        class = "sysevent.fs.zfs.history_event"
        pool = "store1"
        pool_guid = 0x4ab56a61b1481ffb
        pool_state = 0x0
        pool_context = 0x0
        history_hostname = "myhost"
        history_internal_str = "errors=0"
        history_internal_name = "starting deferred resilver"
        history_txg = 0x2b0c09f
        history_time = 0x68c82e40
        time = 0x68c82e40 0x1475246
        eid = 0x328ec4
        
 Sep 15 2025 16:18:24.021451334 sysevent.fs.zfs.resilver_start
        version = 0x0
        class = "sysevent.fs.zfs.resilver_start"
        pool = "store1"
        pool_guid = 0x4ab56a61b1481ffb
        pool_state = 0x0
        pool_context = 0x0
        time = 0x68c82e40 0x1475246
        eid = 0x328ec5

Sep 15 2025 16:18:24.021451334 sysevent.fs.zfs.history_event
        version = 0x0
        class = "sysevent.fs.zfs.history_event"
        pool = "store1"
        pool_guid = 0x4ab56a61b1481ffb
        pool_state = 0x0
        pool_context = 0x0
        history_hostname = "myhost"
        history_internal_str = "func=2 mintxg=3 maxtxg=45136922"
        history_internal_name = "scan setup"
        history_txg = 0x2b0c09f
        history_time = 0x68c82e40
        time = 0x68c82e40 0x1475246
        eid = 0x328ec6       

Redoing the steps of taking enc-dc66-phy10 offline, stopping the scrub process which started just after that, then bringing back the disk online didn't work this time.
Any help or suggestion for dealing with this would be greatly appreciated.

[drook]

Got this on a FreeBSD 15-RC1:

[root@filebackup:~]# uname -a
FreeBSD filebackup 15.0-RC1 FreeBSD 15.0-RC1 releng/15.0-n280957-c87b89d05340 GENERIC amd64
# zpool version
zfs-2.4.0-rc3-FreeBSD_g814f9afba
zfs-kmod-2.4.0-rc3-FreeBSD_g814f9afba