[VirtualKubelet] PVC not reflected on all edge nodes with provision-on-all-edge-nodes annotation

[cheina97]

Background

When a PVC carries the omni.cast.ai/provision-on-all-edge-nodes annotation, every virtual kubelet (VK) should create a corresponding remote PVC on its own cluster. A single local PV is shared across all VKs, with a deterministic name (pvc-), created by whichever VK runs first.

The Bug

Both VKs run the full provisioning flow for provision-on-all-edge-nodes PVCs:

1. Both call ProvisionRemotePVC → each creates a remote PVC on their own cluster ✅
2. Both try to Create the local PV — same deterministic name, race condition
3. The first VK wins. The second gets an AlreadyExists error, which was treated as a real error
4. Handle on the second VK logged a warning and returned before calling Update (which sets VolumeName)
5. On retry: the first VK had already set VolumeName → Kubernetes bound the PVC → removed annSelectedNode
6. shouldProvision on the second VK found no annSelectedNode → returned false → silent skip

Result: no errors visible on the second VK, but the remote PVC on its cluster was never created.

Fix 1 (controller.go)

When shouldProvisionOnAllEdges=true and PV Create returns AlreadyExists, treat it as success. The other VK created the PV, which is expected — the remote PVC on this cluster was still provisioned successfully.

Fix 2 (controller.go / shouldProvision)

for provision-on-all-edge-nodes PVCs that are already bound (VolumeName set, annSelectedNode gone), check whether the remote PVC exists on this cluster. If not, return true to trigger provisioning. This covers the "missed window" case and also self-heals if a remote PVC is accidentally deleted later.

Fix 3 (remoteprovisioner.go)

buildPvNodeAffinity was calling options.SelectedNode.GetLabels() unconditionally, which panics when SelectedNode is nil — as it is when annSelectedNode is absent (Fix 2 path). Since NodeSelectorOpExists does not use node label values, the label lookup is now scoped to the !shouldProvisionOnAllEdges branch only.