[Draft] Add multi-cluster setup scripts and manifests for local development of FederatedRayCluster

[seanlaii]

Why are these changes needed?

This PR adds a local development environment for the upcoming Federated RayCluster feature. It creates 3 interconnected kind clusters using Cilium ClusterMesh, providing the full bidirectional Pod-to-Pod IP connectivity that Ray requires for cross-cluster head-to-worker and worker-to-worker communication.

Ray's networking model demands more than just exposing the head's GCS port -- the head must connect back to workers (for task scheduling, placement groups, node draining), and workers must communicate directly with each other (for object transfer via ray.get(), actor calls). Cilium ClusterMesh provides this transparently across kind clusters, handling Ray's dynamic port requirements without per-port configuration.

What's included:

• hack/bootstrap-federated-ray-lab.sh -- one-command setup: creates 3 kind clusters, installs Cilium, enables. ClusterMesh, connects all clusters into a full mesh
• hack/smoke-test.sh -- verifies all 6 bidirectional cross-cluster Pod-to-Pod connectivity paths
• hack/cleanup-federated-ray-lab.sh -- tears down everything

This environment can be used for developing and testing the FederatedRayCluster CRD, federation controller, and spec.workerOnly support described in the design proposal.

Related issue number

Checks

• I've made sure the tests are passing.
• Testing Strategy
  • Unit tests
  • Manual tests
  • This PR is not tested :(

[seanlaii]

Result from smoke-test.sh:

$ ./hack/smoke-test.sh

=== Deploying echo pods ===
pod/echo-primary created
pod/echo-member-a created
pod/echo-member-b created
=== Waiting for pods to be ready ===
pod/echo-primary condition met
pod/echo-member-a condition met
pod/echo-member-b condition met
=== Getting Pod IPs ===
  echo-primary  (frc-primary):  10.10.2.146
  echo-member-a (frc-member-a): 10.20.1.37
  echo-member-b (frc-member-b): 10.30.2.7

=== Testing cross-cluster Pod-to-Pod connectivity ===
  frc-primary -> frc-member-a (10.20.1.37:5678): OK (hello-from-member-a)
  frc-primary -> frc-member-b (10.30.2.7:5678): OK (hello-from-member-b)
  frc-member-a -> frc-primary (10.10.2.146:5678): OK (hello-from-primary)
  frc-member-a -> frc-member-b (10.30.2.7:5678): OK (hello-from-member-b)
  frc-member-b -> frc-primary (10.10.2.146:5678): OK (hello-from-primary)
  frc-member-b -> frc-member-a (10.20.1.37:5678): OK (hello-from-member-a)

=== Results: 6 passed, 0 failed ===

SMOKE TEST PASSED: All cross-cluster connectivity checks succeeded.

[EagleLo]

Already confirmed the setup script hack/bootstrap-federated-ray-lab.sh is runnable and verified the results with the hack/smoke-test.sh .