Resource delete permanently stuck in DELETING/"Removing Disk": peer's failing `drbdadm adjust` (attach of a LINSTOR-created backing LV smaller than the live DRBD device) blocks deletion of a different node's resource

[abzholdings]

:

Summary

linstor resource delete <rsc> nodeA never completes for a resource whose replica on nodeA is already a diskless tie-breaker with DISK_REMOVING/DELETE flags, because the delete pipeline ("Prepare resource delete" → "Activating resource if necessary before deletion") dispatches an adjust to the remaining peers, and the adjust on nodeC — the InUse consumer, an open Primary diskless client with a pending disk-add — fails every time. The peer's failure is an attach of a backing LV that LINSTOR itself created smaller than the current DRBD device size, so it can never succeed and the delete retries forever. The error persists across a controller restart, a satellite restart, and a clean reboot of the deleting node (after which the kernel resource on nodeA no longer even exists — drbdsetup status says "No such resource" — yet LINSTOR still cannot delete it).

A manual drbdadm adjust <rsc> inside the nodeC satellite container succeeds (exit 0), which made this confusing to debug: the on-disk /var/lib/linstor.d/<rsc>.res still contains disk none; for nodeC (file timestamp several days old), while the config the satellite generates internally during the delete flow carries disk /dev/ubuntu-vg/<rsc>_00000; (visible as c= in the drbdadm -vvv output below) and fails at attach.

Three resources on this cluster are stuck in exactly this state.

Versions

Component	Version
LINSTOR controller / satellite	1.33.3 (GIT c2189575073cbcc5aa04374f5759b1c742675ec6, built 2026-07-10)
piraeus-operator	v2.10.2
DRBD kernel module	9.3.3
drbd-utils	9.34.3
Kernel / OS	7.0.0-28-generic, Ubuntu 26.04 LTS
Kubernetes	v1.36.3
Client issuing the delete	python linstor client 1.28.1 (API 1.0.4)

Topology / current state

╭─────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ ResourceName                             │ Node        │ Layers       │ Usage  │ Conns │ State                   │
╞═══════════════════════════════════════════════════════════════════════════════════════════════════╡
│ pvc-9b996304-68e2-42a4-8338-2ebe369bf56e │ dc01-wrk-01 │ DRBD,STORAGE │        │ Ok    │ DELETING, Removing Disk │
│ pvc-9b996304-68e2-42a4-8338-2ebe369bf56e │ dc01-wrk-02 │ DRBD,STORAGE │ Unused │ Ok    │ UpToDate                │
│ pvc-9b996304-68e2-42a4-8338-2ebe369bf56e │ dc01-wrk-04 │ DRBD,STORAGE │ InUse  │ Ok    │ Diskless                │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────╯

Flags from linstor -m --output-version v1 resource list:

• dc01-wrk-01 (delete target): DELETE, DISKLESS, DISK_REMOVE_REQUESTED, DISK_REMOVING, DRBD_DISKLESS, TIE_BREAKER, DRBD_DELETE; volume 0 flags DRBD_DELETE, storage pool DfltDisklessStorPool. Kernel resource absent on this node since its reboot.
• dc01-wrk-02: no flags; backing /dev/drbdpool/pvc-9b996304-..._00000, LV 10,493,952 KiB, DRBD usable 10,491,672 KiB.
• dc01-wrk-04 (InUse consumer): no DISKLESS flag — volume 0 is registered in storage pool data with device_path: null, allocated_size_kib: -1 (a disk-add that never completed). Kernel state: role:Primary, disk:Diskless client:yes, open:yes, connection to dc01-wrk-02 Established / peer-disk UpToDate.

Controller error (repeats on every delete attempt; reports 6A740EE6-00000-000000 / -000001 / -000002)

ERROR REPORT 6A740EE6-00000-000000
Version: 1.33.3, Build ID c2189575073cbcc5aa04374f5759b1c742675ec6
Error time: 2026-08-06 04:53:20
Class: DelayedApiRcException  (CtrlResponseUtils.java:187)
Error message: Exceptions have been converted to responses
Error context:
    Deletion of resource 'pvc-9b996304-68e2-42a4-8338-2ebe369bf56e' on node 'dc01-wrk-01' failed
    due to an unhandled exception of type DelayedApiRcException.
Asynchronous stage backtrace:
    (dc01-wrk-04) Failed to adjust DRBD resource pvc-9b996304-68e2-42a4-8338-2ebe369bf56e
    Error has been observed at the following site(s):
        *__checkpoint ? Prepare resource delete
        *__checkpoint ? Activating resource if necessary before deletion
Suppressed exception 1 of 2: ApiRcException (CommonMessageProcessor.java:353)
    Message:     (dc01-wrk-04) Failed to adjust DRBD resource pvc-9b996304-68e2-42a4-8338-2ebe369bf56e
    NumericCode: -4611686018427386906

Reports -000001 (05:49:19) and -000002 (05:50:32) are byte-identical apart from timestamps — the last two were after a clean reboot of dc01-wrk-01.

Satellite error on the peer (dc01-wrk-04) — the actual failure

ERROR REPORT 6A740EDB-C9DF7-000040
Module: Satellite   Node: dc01-wrk-04   Error time: 2026-08-06 06:25:59
Class: ResourceException (DrbdLayer.java:940 'adjustDrbd')
Error message: Failed to adjust DRBD resource pvc-9b996304-68e2-42a4-8338-2ebe369bf56e
Caused by: ExtCmdFailedException (DrbdAdm.java:166)
    The external command 'drbdadm' exited with error code 1
    The full command line executed was:
        drbdadm -vvv adjust pvc-9b996304-68e2-42a4-8338-2ebe369bf56e
    Output:
        drbdmeta 1084 v09 /dev/ubuntu-vg/pvc-9b996304-..._00000 internal repair-md
        drbdmeta 1084 v09 /dev/ubuntu-vg/pvc-9b996304-..._00000 internal apply-al
        drbdsetup attach 1084 /dev/ubuntu-vg/pvc-9b996304-..._00000 /dev/ubuntu-vg/pvc-9b996304-..._00000 internal
            --al-extents=6433 --discard-zeroes-if-aligned=no --rs-discard-granularity=4096
    Error information:
        [ne] minor 1084 (vol:0) disk: r=none c=/dev/ubuntu-vg/pvc-9b996304-68e2-42a4-8338-2ebe369bf56e_00000
        [ne] minor 1084 (vol:0) meta-disk: r=(null) c=internal
        1084: Failure: (111) Low.dev. smaller than requested DRBD-dev. size.
        additional info from kernel:
        Current (diskless) capacity 20983344, cannot attach smaller (20975152) disk
        ... terminated with exit code 10

The same report (with an additional Value of 'quorum' differs: r=majority c=off line in some instances) repeats at 04:55:29, 05:05:31, 05:25:44, 05:38:23, 05:38:27, 05:49:19, 05:50:32, 06:25:59 — and also for two more resources in the identical situation (pvc-113238c3-42c2-462b-be7d-49924244f930, pvc-375a05c2-44ad-45e3-9b01-0492f627a5cf).

Why the attach can never succeed

• Current DRBD device size (from the surviving diskful replica on dc01-wrk-02, VG drbdpool, LV 10,493,952 KiB): 20,983,344 sectors = 10,491,672 KiB.
• LINSTOR-created LV on dc01-wrk-04 (VG ubuntu-vg): 10,489,856 KiB, i.e. usable 20,975,152 sectors = 10,487,576 KiB after internal metadata.
• The two storage pools round LV sizes differently (different VG geometry), so the LV created for the disk-add on wrk-04 is 4 MiB smaller than the peer's, and the kernel — correctly — refuses to attach a smaller lower-level device. LINSTOR keeps retrying the identical attach instead of failing the disk-add or resizing the LV, and the pending failure blocks unrelated operations (deleting the other node's replica).

Why manual drbdadm adjust succeeds while LINSTOR's fails

The on-disk config /var/lib/linstor.d/pvc-9b996304-....res on dc01-wrk-04 (generated 2026-08-01, before the disk-add) still contains for the local node:

    on "dc01-wrk-04"
    {
        volume 0
        {
            disk        none;
            ...
            meta-disk   internal;
            device      minor 1084;
        }
        node-id    2;
    }

so an operator running drbdadm adjust gets a no-op success — while the satellite's internally generated config (the c= values in the -vvv output above) includes the new backing device and fails. This mismatch makes the failure look non-reproducible from the shell.

Timeline (2026-08-06, UTC)

• ~04:34 — linstor-controller and the dc01-wrk-04 satellite restarted (fresh error-report instance IDs). No change.
• 04:53:20 — resource delete pvc-9b996304-... dc01-wrk-01 → controller report -000000; satellite adjust failure on wrk-04 (04:55:29).
• 05:05–05:25 — retries; two more resources in the same state fail identically.
• 05:36:33 — clean reboot of dc01-wrk-01 (the deleting node). After boot the DRBD resource no longer exists in its kernel at all.
• 05:49:19, 05:50:32 — delete retried; identical controller reports -000001/-000002.
• 06:25:59 — satellite still failing the same adjust for all three resources.

What was tried

• controller restart — no change
• satellite restart on the failing peer — no change
• manual drbdadm adjust on the peer — exits 0 (no-op, see above), LINSTOR keeps failing
• clean reboot of the deleting node — no change (resource gone from its kernel, LINSTOR state unchanged)

Expected vs actual

Expected: deleting a replica that is already a diskless TieBreaker with DISK_REMOVING should complete (or be forcible), independent of an unrelated, permanently-failing disk-add on another peer. Alternatively/additionally: (a) when a disk-add's auto-created LV cannot satisfy the current DRBD device size, the disk-add should fail and roll back (delete the LV / clear the pending state) rather than leave the satellite's devMgr failing every run; (b) the controller-level error could surface the underlying attach error instead of "Exceptions have been converted to responses".

Actual: the resource is pinned in DELETING, Removing Disk indefinitely; every delete retry produces the same error-report pair; there is no apparent way to cancel the pending disk-add or force the delete.

Questions

1. Is there a supported way to cancel a pending/failed disk-add (clear DISK_REMOVE_REQUESTED/pending attach) or to force-delete the nodeA replica in this state?
2. Should "Activating resource if necessary before deletion" gate deletion of nodeA's resource on an adjust of nodeC at all, when nodeC's failure is unrelated to the replica being deleted?
3. Should LINSTOR size-check auto-created LVs against the live DRBD device size before scheduling the attach (LV rounding differs across pools/VGs)?

Related (checked before filing — none match this mechanism)

• DELETE opration should do drbdadm down first #432 — same surface error during delete ("Failed to adjust DRBD resource"), different cause/workaround (drbdadm down, which does not apply here: the deleting node's kernel resource is already gone).
• Resources are stuck on DELETING with failed replica #101 — stuck DELETING, but the failing disk is on the deleting node itself.
• Resources stuck on DELETING when one of the nodes is not available #112 — stuck DELETING due to an offline node; all nodes here are online with Conns Ok.
• Failed to adjust DRBD resource on toggle-disk primary operation #84 (closed, 2019) — adjust failure on toggle-disk of a primary, ancient version.

Full error reports (controller 6A740EE6-00000-000000/1/2, satellite 6A740EDB-C9DF7-000040) available on request.

================================================================================