You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
After plugging in a USB CDROM drive into a USB3 port instead of a USB2, I appear to have gained zfs corruption. Which is kind of weird, and dodgy. But LLM agrees with me. And the curiously the zvol that is corrupted is one I haven't knowingly touched for many months.
Testing the same drive in a USB3 port I got a kernel panic. Unfortunately, kernel panic stack trace was not saved, and reluctant to try that again. USB2 port works fine without any dramas.
Problem was first detected because zfs mount doesn't mount good partitions, guessing it stops on first error:
root@miacis:/home/brian/ > zfs mount -av
cannot iterate filesystems: I/O error
The SSD is two "Crucial P3 4TB — model CT4000P3PSSD8" in mirror config. It looks like the bad data is on both mirrors. A scrub did not correct the issue.
I had LLM help me draft the rest of the email. I am hoping this means I will capture the important information. Apologies if it got anything wrong that I missed.
Cannot destroy zvol with corrupted DSL directory metadata — zfs destroy fails with I/O error, does not hang but leaves dataset permanently orphaned
Affected dataset:rpool/images/win11 (a ZVOL, Windows 11 VM disk image, non-critical/not backed up)
Background
An unclean shutdown (caused by a failing external USB optical drive destabilizing the USB subsystem) left rpool/images/win11 with corrupted DSL directory metadata on both mirror legs simultaneously. A scrub completed with 0B repaired / 0 errors found, but the pool-level error log continues to report 1–2 persistent data errors tied to this dataset (objset 10227).
Symptoms
1. Dataset cannot be opened by any tool:
$ zfs destroy rpool/images/win11
cannot open 'rpool/images/win11': I/O error
$ zfs destroy -f rpool/images/win11
cannot open 'rpool/images/win11': I/O error
$ zfs destroy -r rpool/images/win11
cannot open 'rpool/images/win11': I/O error
2. Tried zfs_recover=1 (module parameter) before retrying destroy — no change, same clean I/O error.
3. zpool status -v rpool and zpool status -x both hang indefinitely in zio_wait(), confirmed via kernel stack trace:
INFO: task z_rd_int_3:780 is blocked on a mutex likely owned by task zpool:8556.
task:zpool state:D
Call Trace:
zio_wait+0x134/0x2c0 [zfs]
arc_read+0xbcd/0x16a0 [zfs]
dmu_objset_open_impl+0x11b/0xa60 [zfs]
dmu_objset_from_ds+0xf5/0x180 [zfs]
find_birth_txg+0x3f/0x180 [zfs]
check_filesystem+0xcc/0x570 [zfs]
process_error_block+0xa3/0x2c0 [zfs]
process_error_list+0xfa/0x110 [zfs]
spa_get_errlog+0x133/0x1a0 [zfs]
zfs_ioc_error_log+0x54/0x90 [zfs]
This hangs holding spa_namespace_lock, blocking all subsequent zpool/zfs commands system-wide until a hard reboot. This happened twice during troubleshooting.
4. zdb -e -d rpool (pointing directly at both mirror leg device paths) succeeds and cleanly enumerates all other datasets, confirming pool-wide health is otherwise intact:
failed to hold dataset 'rpool/images/win11': Input/output error
[... all other datasets list successfully ...]
MOS object 10224 (DSL directory) leaked
MOS object 10225 (DSL props) leaked
MOS object 10226 (DSL directory child map) leaked
MOS object 10227 (zap) leaked
MOS object 10228 (DSL dataset snap map) leaked
MOS object 10229 (DSL deadlist map) leaked
edonr feature refcount mismatch: 9 consumers != 10 refcount
zstd_compress feature refcount mismatch: 8 consumers != 9 refcount
Questions
Is the zol unrecoverable? LLM seems to think so. But would like to confirm. There was nothing important here except working Windows install with same crap, but I don't like loosing data :-)
Is there a safe way to force-remove a dataset whose DSL directory objects are corrupted on all mirror legs, without needing to open/hold it (i.e., bypassing whatever destroy does internally that requires a successful open)?
Given the zio_wait() hang specifically originates in spa_get_errlog → process_error_block → check_filesystem, is this a known bug where the error-log resolution path should fail fast (like plain destroy does) instead of blocking indefinitely?
Is it safe to leave these MOS objects permanently leaked, or could this cause further issues over time (e.g., during future scrubs, sends, or pool upgrades)?
Happy to provide full zdb -e output, zpool history, or anything else useful. Pool is otherwise healthy and in production use, so I'd like to avoid further risky manual intervention without guidance.
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
After plugging in a USB CDROM drive into a USB3 port instead of a USB2, I appear to have gained zfs corruption. Which is kind of weird, and dodgy. But LLM agrees with me. And the curiously the zvol that is corrupted is one I haven't knowingly touched for many months.
Testing the same drive in a USB3 port I got a kernel panic. Unfortunately, kernel panic stack trace was not saved, and reluctant to try that again. USB2 port works fine without any dramas.
Problem was first detected because zfs mount doesn't mount good partitions, guessing it stops on first error:
The SSD is two "Crucial P3 4TB — model CT4000P3PSSD8" in mirror config. It looks like the bad data is on both mirrors. A scrub did not correct the issue.
I had LLM help me draft the rest of the email. I am hoping this means I will capture the important information. Apologies if it got anything wrong that I missed.
Cannot destroy zvol with corrupted DSL directory metadata —
zfs destroyfails with I/O error, does not hang but leaves dataset permanently orphanedEnvironment
rpool, mirror-0 (2x NVMe, mirrored)rpool/images/win11(a ZVOL, Windows 11 VM disk image, non-critical/not backed up)Background
An unclean shutdown (caused by a failing external USB optical drive destabilizing the USB subsystem) left
rpool/images/win11with corrupted DSL directory metadata on both mirror legs simultaneously. A scrub completed with0B repaired/0 errorsfound, but the pool-level error log continues to report 1–2 persistent data errors tied to this dataset (objset 10227).Symptoms
1. Dataset cannot be opened by any tool:
2. Tried
zfs_recover=1(module parameter) before retrying destroy — no change, same clean I/O error.3.
zpool status -v rpoolandzpool status -xboth hang indefinitely inzio_wait(), confirmed via kernel stack trace:This hangs holding
spa_namespace_lock, blocking all subsequentzpool/zfscommands system-wide until a hard reboot. This happened twice during troubleshooting.4.
zdb -e -d rpool(pointing directly at both mirror leg device paths) succeeds and cleanly enumerates all other datasets, confirming pool-wide health is otherwise intact:Questions
destroydoes internally that requires a successful open)?zio_wait()hang specifically originates inspa_get_errlog→process_error_block→check_filesystem, is this a known bug where the error-log resolution path should fail fast (like plaindestroydoes) instead of blocking indefinitely?Happy to provide full
zdb -eoutput,zpool history, or anything else useful. Pool is otherwise healthy and in production use, so I'd like to avoid further risky manual intervention without guidance.All reactions