e2e: add volume expansion tests for NVMe-oF by nixpanic · Pull Request #6058 · ceph/ceph-csi

nixpanic · 2026-02-13T11:20:12Z

NVMe-oF supports Volume Expansion, but there was no e2e test for it yet.

In order to test this, the following steps are done:

PR includes update to Ceph Tentacle (will cause failure for CephFS)
NVMe-F e2e tests are run (job)
Ceph Tentacle update is removed from this PR
Ready for review. No e2e needed, as NVMe-oF only run on Ceph Tentacle or newer

Show available bot commands

These commands are normally not required, but in case of issues, leave any of
the following bot commands in an otherwise empty comment in this PR:

/retest ci/centos/<job-name>: retest the <job-name> after unrelated
failure (please report the failure too!)

nixpanic · 2026-02-13T11:20:35Z

/test ci/centos/mini-e2e/k8s-1.35/nvmeof

nixpanic · 2026-02-13T12:17:40Z

/test ci/centos/mini-e2e/k8s-1.35/nvmeof

nixpanic · 2026-02-13T13:16:43Z

/test ci/centos/mini-e2e/k8s-1.35/nvmeof

nixpanic · 2026-02-13T13:16:50Z

/test ci/centos/mini-e2e/k8s-1.35/rbd

nixpanic · 2026-02-13T16:14:22Z

/test ci/centos/mini-e2e/k8s-1.35/nvmeof

nixpanic · 2026-02-13T16:14:29Z

/test ci/centos/mini-e2e/k8s-1.35/rbd

gadididi

LGTM!

e2e/nvmeof.go

nixpanic · 2026-02-16T10:48:55Z

/test ci/centos/mini-e2e/k8s-1.35/nvmeof

Pull request has been modified.

nixpanic · 2026-02-16T12:41:03Z

Deleting the NVMe-oF volume failed. A small bug has been introduced with the DH-CHAP feature:

  I0216 11:04:08.599664       1 utils.go:350] ID: 16 Req-ID: 0001-0024-9011eb60-3b65-4891-8d1f-846868ee065d-0000000000000007-4ad3ec9f-c916-4b67-ade3-d8d5e09caa93 GRPC call: /csi.v1.Controller/DeleteVolume
  I0216 11:04:08.599742       1 utils.go:351] ID: 16 Req-ID: 0001-0024-9011eb60-3b65-4891-8d1f-846868ee065d-0000000000000007-4ad3ec9f-c916-4b67-ade3-d8d5e09caa93 GRPC request: {"secrets":"***stripped***","volume_id":"0001-0024-9011eb60-3b65-4891-8d1f-846868ee065d-0000000000000007-4ad3ec9f-c916-4b67-ade3-d8d5e09caa93"}
  I0216 11:04:08.601453       1 omap.go:89] ID: 16 Req-ID: 0001-0024-9011eb60-3b65-4891-8d1f-846868ee065d-0000000000000007-4ad3ec9f-c916-4b67-ade3-d8d5e09caa93 got omap values: (pool="nvmeofpool", namespace="", name="csi.volume.4ad3ec9f-c916-4b67-ade3-d8d5e09caa93"): map[csi.imageid:124e7c5e59c0 csi.imagename:csi-vol-4ad3ec9f-c916-4b67-ade3-d8d5e09caa93 csi.volname:pvc-e73a51b1-fbc8-4191-bd57-e774ae1a7000 csi.volume.owner:nvmeof-4145] 
  I0216 11:04:08.691558       1 controllerserver.go:211] ID: 16 Req-ID: 0001-0024-9011eb60-3b65-4891-8d1f-846868ee065d-0000000000000007-4ad3ec9f-c916-4b67-ade3-d8d5e09caa93 No NVMe-oF metadata found, skipping NVMe-oF cleanup: metadata not found: failed to get .rbd.nvmeof.dhchapMode: RBD image not found: rbd: ret=-2, No such file or directory
  I0216 11:04:08.692639       1 omap.go:89] ID: 16 Req-ID: 0001-0024-9011eb60-3b65-4891-8d1f-846868ee065d-0000000000000007-4ad3ec9f-c916-4b67-ade3-d8d5e09caa93 got omap values: (pool="nvmeofpool", namespace="", name="csi.volume.4ad3ec9f-c916-4b67-ade3-d8d5e09caa93"): map[csi.imageid:124e7c5e59c0 csi.imagename:csi-vol-4ad3ec9f-c916-4b67-ade3-d8d5e09caa93 csi.volname:pvc-e73a51b1-fbc8-4191-bd57-e774ae1a7000 csi.volume.owner:nvmeof-4145] 
  E0216 11:04:08.731204       1 controllerserver.go:1091] ID: 16 Req-ID: 0001-0024-9011eb60-3b65-4891-8d1f-846868ee065d-0000000000000007-4ad3ec9f-c916-4b67-ade3-d8d5e09caa93 rbd nvmeofpool/csi-getNVMeoFMetadatavol-4ad3ec9f-c916-4b67-ade3-d8d5e09caa93 is still being used
  E0216 11:04:08.731268       1 utils.go:355] ID: 16 Req-ID: 0001-0024-9011eb60-3b65-4891-8d1f-846868ee065d-0000000000000007-4ad3ec9f-c916-4b67-ade3-d8d5e09caa93 GRPC error: rpc error: code = Aborted desc = rbd csi-vol-4ad3ec9f-c916-4b67-ade3-d8d5e09caa93 is still being used

Because getNVMeoFMetadata returns an error, the RBD-image is not removed from the gateway, so it is really still in use.

@gadididi has reviewed the issue and will provide a fix for it.

gadididi · 2026-02-16T13:12:14Z

/test ci/centos/mini-e2e/k8s-1.35/nvmeof

gadididi · 2026-02-17T14:52:44Z

/test ci/centos/mini-e2e/k8s-1.35/nvmeof

gadididi · 2026-02-17T16:21:56Z

/test ci/centos/mini-e2e/k8s-1.35/nvmeof

gadididi · 2026-02-17T18:41:21Z

I opened fix PR for the dh-chap (#6085) because that bug in not belong only to this PR
and I think it is better this fix will be merge ASAP because it critical issue (on deleting the volume)

gadididi · 2026-03-22T13:54:59Z

/test ci/centos/mini-e2e/k8s-1.35/nvmeof

e2e/ceph.go

Tests for filesystem expansion and block-mode expansion are added. To make the `resizePVCAndValidateSize()` helper a little more flexible, it has been adapted to get a filename or PVC-object as 1st parameter. Signed-off-by: Niels de Vos <ndevos@ibm.com>

add gw logs for e2e test log file. Signed-off-by: gadi-didi <gadi.didi@ibm.com>

Signed-off-by: Niels de Vos <ndevos@ibm.com>

gadididi · 2026-03-30T05:10:06Z

https://github.com/Mergifyio rebase

mergify · 2026-03-30T05:10:20Z

rebase

☑️ Nothing to do, the required conditions are not met

Details

any of:
- #commits-behind > 0 [📌 rebase requirement]
- -linear-history [📌 rebase requirement]
-closed [📌 rebase requirement]
-conflict [📌 rebase requirement]
queue-position = -1 [📌 rebase requirement]

ceph-csi-bot · 2026-03-30T08:07:12Z

/test ci/centos/upgrade-tests-cephfs

ceph-csi-bot · 2026-03-30T08:07:12Z

/test ci/centos/k8s-e2e-external-storage/1.34

ceph-csi-bot · 2026-03-30T08:07:12Z

/test ci/centos/k8s-e2e-external-storage/1.33

ceph-csi-bot · 2026-03-30T08:07:12Z

/test ci/centos/upgrade-tests-rbd

ceph-csi-bot · 2026-03-30T08:07:13Z

/test ci/centos/mini-e2e-helm/k8s-1.34

ceph-csi-bot · 2026-03-30T08:07:13Z

/test ci/centos/mini-e2e-helm/k8s-1.33

ceph-csi-bot · 2026-03-30T08:07:13Z

/test ci/centos/k8s-e2e-external-storage/1.35

ceph-csi-bot · 2026-03-30T08:07:13Z

/test ci/centos/mini-e2e/k8s-1.34

ceph-csi-bot · 2026-03-30T08:07:13Z

/test ci/centos/mini-e2e/k8s-1.33

ceph-csi-bot · 2026-03-30T08:07:13Z

/test ci/centos/mini-e2e-helm/k8s-1.35

ceph-csi-bot · 2026-03-30T08:07:14Z

/test ci/centos/mini-e2e/k8s-1.35

mergify · 2026-03-30T11:14:27Z

Merge Queue Status

✅ Entered queue — 2026-03-30 11:14 UTC · Rule: default
✅ Checks skipped · PR is already up-to-date
✅ Merged — 2026-03-30 11:14 UTC · at d7f9f244a4dc93840c1fe85c967b412da68da000

This pull request spent 14 seconds in the queue, including 2 seconds running CI.

Required conditions to merge

#approved-reviews-by >= 2 [🛡 GitHub branch protection]
- e2e: add volume expansion tests for NVMe-oF #6058
#changes-requested-reviews-by = 0 [🛡 GitHub branch protection]
- e2e: add volume expansion tests for NVMe-oF #6058

nixpanic added ci/skip/multi-arch-build skip building on multiple architectures component/nvme-of Issues and PRs related to NVMe-oF. labels Feb 13, 2026

mergify bot added the component/testing Additional test cases or CI work label Feb 13, 2026

nixpanic force-pushed the nvmeof/e2e/volume-expand branch from 71b83d3 to 3f651a0 Compare February 13, 2026 12:17

nixpanic force-pushed the nvmeof/e2e/volume-expand branch from 3f651a0 to f3faa3c Compare February 13, 2026 13:16

nixpanic force-pushed the nvmeof/e2e/volume-expand branch from f3faa3c to 070ac74 Compare February 13, 2026 16:14

nixpanic force-pushed the nvmeof/e2e/volume-expand branch from 070ac74 to 77b03fb Compare February 13, 2026 16:45

nixpanic added the ci/skip/e2e skip running e2e CI jobs label Feb 13, 2026

nixpanic requested a review from a team February 13, 2026 16:45

nixpanic marked this pull request as ready for review February 13, 2026 16:45

nixpanic requested a review from gadididi February 13, 2026 16:46

gadididi previously approved these changes Feb 16, 2026

View reviewed changes

Madhu-1 reviewed Feb 16, 2026

View reviewed changes

e2e/nvmeof.go Outdated Show resolved Hide resolved

nixpanic force-pushed the nvmeof/e2e/volume-expand branch from 77b03fb to 1fa5ae4 Compare February 16, 2026 10:48

nixpanic removed the ci/skip/e2e skip running e2e CI jobs label Feb 16, 2026

gadididi force-pushed the nvmeof/e2e/volume-expand branch from 7e238fb to a48627a Compare February 16, 2026 13:08

nixpanic marked this pull request as draft February 16, 2026 16:29

nixpanic requested review from a team and gadididi March 20, 2026 09:31

nixpanic force-pushed the nvmeof/e2e/volume-expand branch from 3720606 to 6f8d3ef Compare March 20, 2026 09:47

black-dragon74 reviewed Mar 27, 2026

View reviewed changes

e2e/ceph.go Show resolved Hide resolved

nixpanic and others added 3 commits March 27, 2026 14:03

e2e: include the NVMe-oF gateway logs after a failure

6c09218

add gw logs for e2e test log file. Signed-off-by: gadi-didi <gadi.didi@ibm.com>

e2e: cleanup Ceph version API for easier comparing

d7f9f24

Signed-off-by: Niels de Vos <ndevos@ibm.com>

nixpanic force-pushed the nvmeof/e2e/volume-expand branch from 6f8d3ef to d7f9f24 Compare March 27, 2026 13:04

nixpanic requested a review from black-dragon74 March 27, 2026 13:04

black-dragon74 approved these changes Mar 27, 2026

View reviewed changes

nixpanic requested a review from a team March 27, 2026 13:42

Madhu-1 approved these changes Mar 30, 2026

View reviewed changes

nixpanic added the ok-to-test Label to trigger E2E tests label Mar 30, 2026

ceph-csi-bot removed the ok-to-test Label to trigger E2E tests label Mar 30, 2026

mergify bot merged commit 761092e into ceph:devel Mar 30, 2026
38 checks passed

Conversation

nixpanic commented Feb 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nixpanic commented Feb 13, 2026

Uh oh!

nixpanic commented Feb 13, 2026

Uh oh!

nixpanic commented Feb 13, 2026

Uh oh!

nixpanic commented Feb 13, 2026

Uh oh!

nixpanic commented Feb 13, 2026

Uh oh!

nixpanic commented Feb 13, 2026

Uh oh!

gadididi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

nixpanic commented Feb 16, 2026

Uh oh!

nixpanic commented Feb 16, 2026

Uh oh!

gadididi commented Feb 16, 2026

Uh oh!

gadididi commented Feb 17, 2026

Uh oh!

gadididi commented Feb 17, 2026

Uh oh!

gadididi commented Feb 17, 2026

Uh oh!

gadididi commented Mar 22, 2026

Uh oh!

Uh oh!

gadididi commented Mar 30, 2026

Uh oh!

mergify bot commented Mar 30, 2026

☑️ Nothing to do, the required conditions are not met

Uh oh!

ceph-csi-bot commented Mar 30, 2026

Uh oh!

ceph-csi-bot commented Mar 30, 2026

Uh oh!

ceph-csi-bot commented Mar 30, 2026

Uh oh!

ceph-csi-bot commented Mar 30, 2026

Uh oh!

ceph-csi-bot commented Mar 30, 2026

Uh oh!

ceph-csi-bot commented Mar 30, 2026

Uh oh!

ceph-csi-bot commented Mar 30, 2026

Uh oh!

ceph-csi-bot commented Mar 30, 2026

Uh oh!

ceph-csi-bot commented Mar 30, 2026

Uh oh!

ceph-csi-bot commented Mar 30, 2026

Uh oh!

ceph-csi-bot commented Mar 30, 2026

Uh oh!

mergify bot commented Mar 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merge Queue Status

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

nixpanic commented Feb 13, 2026 •

edited

Loading

mergify bot commented Mar 30, 2026 •

edited

Loading