Slow import when rebooting with MMP after upgrading to 2.4.1

[stuartthebruce]

System information

Type	Version/Name
Distribution Name	Rock Linux
Distribution Version	9.7
Kernel Version	5.14.0-611.36.1
Architecture	x86_64
OpenZFS Version	2.4.1

Describe the problem you're observing

After upgrading from 2.4.0 to 2.4.1 the time to import a pool with multihost=on from an external JBOD with 60 HDD has increased to ~4 minutes. I suspect this is due to #18177, but it would be nice if that extra safety could be optimized to reduce system boot time.

Describe how to reproduce the problem

Reboot a system with qty 8 external 60-drive JBOD pools as 8 separate zpool with multihost=on and wait for zfs-import-cache.service to add ~30 minutes to the boot time.

Include any warning/errors/backtraces from the system logs

[root@zfsarchive1 ~]# systemctl status zfs-import-cache.service
● zfs-import-cache.service - Import ZFS pools by cache file
     Loaded: loaded (/usr/lib/systemd/system/zfs-import-cache.service; enabled; preset: enabled)
     Active: active (exited) since Tue 2026-03-17 10:41:42 PDT; 6 days ago
       Docs: man:zpool(8)
   Main PID: 31263 (code=exited, status=0/SUCCESS)
        CPU: 37.298s

Mar 17 10:10:25 zfsarchive1 systemd[1]: Starting Import ZFS pools by cache file...
Mar 17 10:41:42 zfsarchive1 systemd[1]: Finished Import ZFS pools by cache file.

[tonyhutter]

I assume both nodes with access to the pool have /etc/hostid setup? (https://openzfs.github.io/openzfs-docs/man/master/8/zgenhostid.8.html)

[stuartthebruce]

I assume both nodes with access to the pool have /etc/hostid setup? (https://openzfs.github.io/openzfs-docs/man/master/8/zgenhostid.8.html)

The example host does indeed have /etc/hostid set, but there where no other hosts physically connected to the JBODs in question during the above measurement, which has been reproduced in another server connected to a different set of 8 JBOD. We have multihost=on because there have been (and may likely again be) multiple hosts connected to many of the JBODs.

[tonyhutter]

Could you possibly attach the output from /proc/spl/kstat/zfs/dbgmsg? That will have the multihost debug messages.

[stuartthebruce]

Could you possibly attach the output from /proc/spl/kstat/zfs/dbgmsg? That will have the multihost debug messages.

Will do so latter today after this system comes back from a scheduled building power outage.

[stuartthebruce]

dbgmsg.zfs.gz

[stuartthebruce]

In addition to possible optimizations for individual pool import is there also a possibility for parallel imports at boot time?

[tonyhutter]

I see in your logs that it's doing the activity check:

1774397011   ff18253683ef8000 spa_misc.c:2489:spa_import_progress_set_notes_impl(): 'jbod18' Checking MMP activity, waiting 207200 ms

207200 ms = ~20min.

The activity check will normally be bypassed if you cleanly export the pool and keep the hostid on the node the same across imports (see spa_activity_check_required()):

zfs/module/zfs/spa.c

Line 3868 in b44a3ec

spa_activity_check_required(spa_t *spa, uberblock_t *ub, nvlist_t *label)

Do you know if the pool is being exported cleanly? Is hostid consistent across imports?

[stuartthebruce]

Do you know if the pool is being exported cleanly?

This is happening after a clean /sbin/shutdown -r now, where "clean" means no manual intervention required for the system to shutdown and reboot; in addition zfs-import-cache.service is is green after the reboot. I will time a single pool export and import time to remove any systemd complexity.

Is hostid consistent across imports?

Yes.

[stuartthebruce]

The time to import after a clean export is only ~5 seconds per pool,

[root@zfsarchive1 ~]# time zpool export jbod18
real    0m1.721s
user    0m0.003s
sys     0m0.414s

[root@zfsarchive1 ~]# time zpool import jbod18
real    0m5.034s
user    0m1.601s
sys     0m3.479s

Does this imply that the standard EL9 systemd shutdown process on these NFS servers is leaving some residue that triggers an unnecessary long running safety check when importing at boot time?

Let me look around to see if I can find another less critical large HDD jbod system to try a few more tests on.

[tonyhutter]

Does this imply that the standard EL9 systemd shutdown process on these NFS servers is leaving some residue that triggers an unnecessary long running safety check when importing at boot time?

It's possible the pool isn't getting exported or hanging on export, but the shutdown continues anyway.

[mschilli87]

@stuartthebruce: Would you mind adjusting the issue title to reflect that new information (and your uncertainty regarding what is actually going wrong and where)? Right now, a drive-by user might get the (potentially false) impression that upgrading to 2.4.1 causes a major regression.

[stuartthebruce]

I have reproduced this issue (~4 min import time after a reboot for a 60 HDD pool) on additional RL8 and AL10 systems that each have a single JBOD/zpool but are also NFS servers.

[tonyhutter]

I have reproduced this issue (~4 min import time after a reboot for a 60 HDD pool) on additional RL8 and AL10 systems that each have a single JBOD/zpool but are also NFS servers.

Is the 4min import after a manual export/import or an import after shutdown?