nfs: implement ControllerModifyVolume to change NFS-server hostname #5829

nixpanic · 2025-12-04T16:52:38Z

The new hostname of the NFS-server is stored in a journal entry for the
volume. The new NFS-server will only be used the next time the NFS
Volume is mounted on a worker node.

It is not possible to update the VolumeContext of a Volume, so the old
hostname of the NFS-server will be stored there forever.

Show available bot commands

These commands are normally not required, but in case of issues, leave any of
the following bot commands in an otherwise empty comment in this PR:

/retest ci/centos/<job-name>: retest the <job-name> after unrelated
failure (please report the failure too!)

Madhu-1

@nixpanic is this tested code? i dont see ControllerServiceCapability_RPC_MODIFY_VOLUME capability is exposed for nFS

nixpanic · 2025-12-10T14:00:01Z

@nixpanic is this tested code? i dont see ControllerServiceCapability_RPC_MODIFY_VOLUME capability is exposed for nFS

Still trying to add some e2e tests, the csi-provisioner and csi-resizer sidecars have just gained the ability to pass credentials to the ControllerModifyVolume request.

Pull request has been modified.

nixpanic · 2025-12-12T10:20:06Z

/test ci/centos/mini-e2e-helm/k8s-1.34/nfs

nixpanic · 2025-12-12T10:48:09Z

/test ci/centos/mini-e2e/k8s-1.34/nfs

nixpanic · 2025-12-12T11:25:38Z

/test ci/centos/mini-e2e/k8s-1.34

nixpanic · 2025-12-12T12:32:59Z

@nixpanic is this tested code? i dont see ControllerServiceCapability_RPC_MODIFY_VOLUME capability is exposed for nFS

Verified 😜 that it fails with this:

  I1212 12:26:38.595866 75590 pvc.go:85] PVC cephcsi-nfs-pvc Event: ProvisioningFailed - CSI driver does not support VolumeAttributesClass: controller MODIFY_VOLUME capability is not reported

nixpanic · 2025-12-12T12:35:43Z

/test ci/centos/mini-e2e/k8s-1.34

nixpanic · 2025-12-12T12:50:48Z

/test ci/centos/mini-e2e/k8s-1.34

nixpanic · 2025-12-12T13:10:08Z

/test ci/centos/mini-e2e/k8s-1.34

nixpanic · 2025-12-16T08:48:59Z

/test ci/centos/mini-e2e/k8s-1.34/cephfs

nixpanic · 2025-12-16T15:31:44Z

/test ci/centos/mini-e2e/k8s-1.34/cephfs

nixpanic · 2025-12-17T09:09:06Z

/test ci/centos/mini-e2e/k8s-1.34/cephfs

nixpanic · 2025-12-17T12:19:59Z

/test ci/centos/mini-e2e/k8s-1.34/cephfs

nixpanic · 2025-12-19T13:15:28Z

Still trying to use the old nfs-server:

  Warning  FailedMount  8s (x13 over 10m)  kubelet            MountVolume.SetUp failed for volume "pvc-c64d91d7-9097-46f3-9594-eb993e5851b7" : rpc error: code = Internal desc = nfs: failed to mount "relocated.example.net:/0001-0024-e044d94f-5dfb-4f7b-a7e6-44895f196074-0000000000000001-38cbb13b-1f15-4a9b-80ff-3017574691a1" to "/var/lib/kubelet/pods/4ebe7989-0889-4af1-b138-48c67a90debd/volumes/kubernetes.io~csi/pvc-c64d91d7-9097-46f3-9594-eb993e5851b7/mount" : mount failed: exit status 32
Mounting command: mount
Mounting arguments: -t nfs relocated.example.net:/0001-0024-e044d94f-5dfb-4f7b-a7e6-44895f196074-0000000000000001-38cbb13b-1f15-4a9b-80ff-3017574691a1 /var/lib/kubelet/pods/4ebe7989-0889-4af1-b138-48c67a90debd/volumes/kubernetes.io~csi/pvc-c64d91d7-9097-46f3-9594-eb993e5851b7/mount
Output: mount.nfs: Failed to resolve server relocated.example.net: Name or service not known
 stderr: ""

nixpanic · 2025-12-19T13:43:59Z

It seems I missed handling of mutable_parameters in CreateVolume CSI procedure.

nixpanic · 2025-12-19T15:55:19Z

/test ci/centos/mini-e2e/k8s-1.34/cephfs

nixpanic · 2026-01-06T09:30:52Z

/test ci/centos/mini-e2e/k8s-1.34/cephfs

Pull request has been modified.

nixpanic · 2026-02-03T10:55:33Z

It seems there has been an update to the csi-resizer:canary image and watch is required now too:

  E0201 19:25:29.550775       1 reflector.go:205] "Failed to watch" err="volumeattributesclasses.storage.k8s.io is forbidden: User \"system:serviceaccount:cephcsi-e2e-ae4861ac:nfs-csi-provisioner\" cannot watch resource \"volumeattributesclasses\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.VolumeAttributesClass"

Updated the RBAC for the provisioner, please have a look again.

nixpanic · 2026-02-03T10:55:55Z

/test ci/centos/mini-e2e/k8s-1.34

nixpanic · 2026-02-04T09:55:23Z

strange, it looks like these StorageClass keys are not accepted, but those were added to the external-provisioner 🤔

csi.storage.k8s.io/controller-modify-secret-name
csi.storage.k8s.io/controller-modify-secret-namespace

The logs show

  I0203 13:29:35.730236       1 csi-provisioner.go:170] "Version" version="v6.0.0"

Not sure that it is the canary image mentioned in build.env. Even stranger is that this has passed CI before, and inspection of the logs showed correct functioning.

Signed-off-by: Niels de Vos <[email protected]>

All functions use a fail-early-continue-on-success flow, NodeGetVolumeStats was not following this. Signed-off-by: Niels de Vos <[email protected]>

Signed-off-by: Niels de Vos <[email protected]>

nixpanic · 2026-02-04T09:58:07Z

/test ci/centos/mini-e2e/k8s-1.34

nixpanic · 2026-02-04T10:04:31Z

Not sure that it is the canary image mentioned in build.env. Even stranger is that this has passed CI before, and inspection of the logs showed correct functioning.

strange, it looks like these StorageClass keys are not accepted, but those were added to the external-provisioner 🤔
* csi.storage.k8s.io/controller-modify-secret-name

* csi.storage.k8s.io/controller-modify-secret-namespace
The logs show
  I0203 13:29:35.730236       1 csi-provisioner.go:170] "Version" version="v6.0.0"
Not sure that it is the canary image mentioned in build.env. Even stranger is that this has passed CI before, and inspection of the logs showed correct functioning.

Added the canary image in commit "Testing: use canary images for csi-resizer/provisioner", not only in build.env, but also under deploy/..../nfs.

The new hostname of the NFS-server is stored in a journal entry for the volume. The new NFS-server will only be used the next time the NFS Volume is mounted on a worker node. It is not possible to update the VolumeContext of a Volume, so the old hostname of the NFS=server will be stored there forever. Closes: ceph#5420 Signed-off-by: Niels de Vos <[email protected]>

Signed-off-by: Niels de Vos <[email protected]>

Only the canary version suppors the VolumeAttributesClass with secrets in the ControllerModifyVolume gRPC. Signed-off-by: Niels de Vos <[email protected]>

Signed-off-by: Niels de Vos <[email protected]>

The NFS node-plugin does not support staging, it only uses publishing. For VolumeAttributeClass / ControllerModifyVolume the mutable parameters are stored in the Ceph backend. This means that during volume publishing the node-plugin needs to get the updated parameters from the Ceph cluster (hence the publish secret requirement). /tmp/csi has been added to the node-plugin Pod so that the temporary credentials file can be written. Signed-off-by: Niels de Vos <[email protected]>

Signed-off-by: Niels de Vos <[email protected]>

Only Kubernetes 1.34 and newer support VolumeAttributeClass functionality (GA). Ceph-CSI also depends on certain versions of the Kubernetes CSI external-provisioner and external-resizer. See-also: kubernetes-csi/docs#631 Signed-off-by: Niels de Vos <[email protected]>

nixpanic · 2026-02-04T11:41:16Z

/test ci/centos/mini-e2e/k8s-1.34

mergify bot added the component/nfs Issues related to NFS label Dec 4, 2025

nixpanic force-pushed the nfs/ControllerModifyVolume branch from d006863 to 1a11979 Compare December 4, 2025 17:08

Rakshith-R previously approved these changes Dec 10, 2025

View reviewed changes

Madhu-1 reviewed Dec 10, 2025

View reviewed changes

nixpanic marked this pull request as draft December 10, 2025 14:00

nixpanic force-pushed the nfs/ControllerModifyVolume branch from 1a11979 to ca55fc1 Compare December 12, 2025 10:18

nixpanic force-pushed the nfs/ControllerModifyVolume branch from ca55fc1 to fe5d071 Compare December 12, 2025 12:35

nixpanic mentioned this pull request Dec 16, 2025

rebase: bump the k8s-dependencies group in /e2e with 4 updates #5864

Merged

nixpanic force-pushed the nfs/ControllerModifyVolume branch from 0215938 to fc84b94 Compare December 16, 2025 15:31

nixpanic force-pushed the nfs/ControllerModifyVolume branch from fc84b94 to df7f5dd Compare December 17, 2025 09:08

nixpanic force-pushed the nfs/ControllerModifyVolume branch from df7f5dd to 0f12bea Compare December 17, 2025 12:19

nixpanic force-pushed the nfs/ControllerModifyVolume branch from 0f12bea to dc81558 Compare December 19, 2025 15:55

nixpanic force-pushed the nfs/ControllerModifyVolume branch from dc81558 to 8899dcc Compare January 6, 2026 13:24

mergify bot added dequeued and removed queued labels Feb 1, 2026

nixpanic force-pushed the nfs/ControllerModifyVolume branch from ea99e05 to 29196c5 Compare February 3, 2026 10:54

mergify bot removed the dequeued label Feb 3, 2026

nixpanic requested review from a team, Madhu-1 and Rakshith-R February 3, 2026 10:56

nixpanic added 4 commits February 4, 2026 10:57

nfs: create helpers for storing volume attributes in the journal

c840ac7

Signed-off-by: Niels de Vos <[email protected]>

nfs: fail early in NodeGetVolumeStats

7584ff3

All functions use a fail-early-continue-on-success flow, NodeGetVolumeStats was not following this. Signed-off-by: Niels de Vos <[email protected]>

nfs: move NFSVolume to shared types package

b5334e8

Signed-off-by: Niels de Vos <[email protected]>

nfs: make paramServer shared option public

f546028

Signed-off-by: Niels de Vos <[email protected]>

nixpanic force-pushed the nfs/ControllerModifyVolume branch from 29196c5 to cd729f3 Compare February 4, 2026 09:57

nixpanic added 9 commits February 4, 2026 12:40

e2e: cleanup k8s.io module dependencies

9dc3362

Signed-off-by: Niels de Vos <[email protected]>

e2e: add VolumeAttributesClass test for NFS

4f514e9

Signed-off-by: Niels de Vos <[email protected]>

doc: include reference to NFS/ControllerModifyVolume in release-notes

dd8b92e

Signed-off-by: Niels de Vos <[email protected]>

testing: use canary images for csi-resizer/provisioner

4ef858a

Only the canary version suppors the VolumeAttributesClass with secrets in the ControllerModifyVolume gRPC. Signed-off-by: Niels de Vos <[email protected]>

deploy: allow NFS-provisioner to access volumeattributesclasses

8547fd6

Signed-off-by: Niels de Vos <[email protected]>

doc: mention support for NFS-server hostname changing

ed84d52

Signed-off-by: Niels de Vos <[email protected]>

nixpanic force-pushed the nfs/ControllerModifyVolume branch from cd729f3 to a38a8cc Compare February 4, 2026 11:40

nfs: implement ControllerModifyVolume to change NFS-server hostname #5829

Are you sure you want to change the base?

nfs: implement ControllerModifyVolume to change NFS-server hostname #5829

Uh oh!

Conversation

nixpanic commented Dec 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Madhu-1 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nixpanic commented Dec 10, 2025

Uh oh!

nixpanic commented Dec 12, 2025

Uh oh!

nixpanic commented Dec 12, 2025

Uh oh!

nixpanic commented Dec 12, 2025

Uh oh!

nixpanic commented Dec 12, 2025

Uh oh!

nixpanic commented Dec 12, 2025

Uh oh!

nixpanic commented Dec 12, 2025

Uh oh!

nixpanic commented Dec 12, 2025

Uh oh!

nixpanic commented Dec 16, 2025

Uh oh!

nixpanic commented Dec 16, 2025

Uh oh!

nixpanic commented Dec 17, 2025

Uh oh!

nixpanic commented Dec 17, 2025

Uh oh!

nixpanic commented Dec 19, 2025

Uh oh!

nixpanic commented Dec 19, 2025

Uh oh!

nixpanic commented Dec 19, 2025

Uh oh!

nixpanic commented Jan 6, 2026

Uh oh!

nixpanic commented Feb 3, 2026

Uh oh!

nixpanic commented Feb 3, 2026

Uh oh!

nixpanic commented Feb 4, 2026

Uh oh!

nixpanic commented Feb 4, 2026

Uh oh!

nixpanic commented Feb 4, 2026

Uh oh!

nixpanic commented Feb 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

nixpanic commented Dec 4, 2025 •

edited

Loading

Madhu-1 left a comment •

edited

Loading