ZFS on LUKS: shutting down devices fails

[grim-ripper]

Hi

I was looking around about these error messages that appear in journalctl:
device-mapper: remove ioctl on [...] failed: Device or resource busy
There are also similar messages about /dev/dm-X devices, appearing during shutdown after some delay. They don't appear to be persisted in any logs though. As far as I understand, these are the same Device Mapper devices for LUKS HDDs. The devices in the journalctl error messages are in /dev/mapper.

I have a zpool on these LUKS HDDs, which likely causes them to remain "busy". Here's some discussions about similar situations:

• https://www.reddit.com/r/zfs/comments/1qcwdd0/zfs_on_top_of_luks_unable_to_cleanly_close_luks/
• systemd-cryptsetup: fails to close encrypted LUKS device on shutdown with zfs #17773
• systemd-cryptsetup: fails to close encrypted LUKS device on shutdown (with zfs) systemd/systemd#39062

Apparently the only way to "close" the zpool so that the LUKS devices can be properly closed is to export the zpool. Which then leads to the necessity of it being scanned again on next startup, because it's removed from import cache. I tried shutting down the NFS server since a filesystem in the zpool is shared, and also zfs unmount, but neither had any effect on the aforementioned error messages. Not surprisingly, since the zpool, as far as I understand, remains active.

Is this correct? The only working way to "close" the zpool so that the devices are no longer "busy" is export, which then needs an explicit import instead of the zpool currently being added back by zfs-import-cache.service? I could add a systemd service to do the export, and apparently using zfs-import-scan.service instead of zfs-import-cache.service would handle importing the zpool after it has been exported.

I'd like confirmation that this is the correct approach, and that ZFS doesn't provide anything out of the box to suitably "close" the zpool and free the devices. I´d also like to know more about what difference this makes in terms of data integrity and such.

[VectorPeak]

Yes - that understanding is correct. For an imported pool, ZFS keeps the vdev block devices open. zfs unmount only unmounts datasets, and stopping NFS only removes one possible user of those datasets; neither tears down the imported pool itself. The normal ZFS operation that releases the underlying LUKS/dm devices is zpool export.

Using a shutdown unit to run something like:

zpool export <poolname>

and then importing via device scan on the next boot is a reasonable approach for this layout, assuming the LUKS devices are opened before the ZFS import runs. zfs-import-cache.service is cache-driven, and an exported pool is no longer treated the same way there; zfs-import-scan.service or an explicit import service for that pool is the usual way around that.

From a data-integrity perspective, zpool export is the cleanest state: pending transactions are synced, filesystems are unmounted, the pool is marked exported, and the vdevs are closed. A normal shutdown without export should still leave ZFS crash-consistent, but it does not satisfy dm-crypt’s requirement that no open holders remain before closing the mapper device.

---

If my answer solved your problem, please mark it as answered the question — I'm here to help, and honestly I'm also collecting Galaxy Brain badges along the way 😆