Non-hcloud nodes and HCCM with private networks

[deubert-it]

We are experimenting with adding k3s nodes to clusters set up by hetzner-k3s, and while HCCM officially supports hetzner cloud nodes as well as hetzner robot nodes, our custom nodes are neither (think generic VMs).

While this is not strictly hetzner-k3s related, as I read hetzner-k3s wants to provide support for "anything with ssh access", it probably is a future problem here as well, so this discussion here might make sense.

The current issue:
Our clusters make use of private networking. HCCM has a route controller that wants to create routes per node to private networks.
Because HCCM can't find our custom node being a hcloud or robot instance, it will constantly fail.

Warning  FailedToCreateRoute      88s (x285756 over 103d)  route_controller  (combined from similar events): Could not create route f8ae0166-fa3a-4c03-9515-d72457a9c856 10.244.8.0/24 for node acme-stage-kube01 after 162.767µs: hcloud/CreateRoute: hcops/AllServersCache.ByName: acme-stage-kube01 hcops/AllServersCache.getCache: not found                                    

While hetzner CSI controller has a simple way to not consider nodes by setting the instance.hetzner.cloud/is-root-server=true label to the node, there is nothing similar for HCCM.

There is an option to fully deactivate the Route Controller for HCCM, but I'm not sure that's good for our hcloud or robot nodes.
For our existing environments, having hybrid/mixed nodes makes a lot of sense, so it's important for us.

However, I don't think our current setup makes use of those routes at all, because as per hetzner documentation it would only work with specific CNI configuration, i.e. cilium with routing_mode: 'native', which we don't use. (we use cilium with mostly standard values, and routing_mode default is 'tunnel').
I'm still analyzing this - if we actually need it somewhere, or if it would have benefits to do so.

We currently see the recurring node events and hccm routecontroller is trying to do this every 30 seconds, which just doesn't feel right.

Maybe I missed something? Also planning on raising a HCCM issue to maybe have some way to not consider some nodes via label/annotation?

[deubert-it]

I found this discussion here which covers adding a ROBOT server, but also ends with just disabling the route controller: #489 (reply in thread)

[vitobotta]

Hi! Thanks for opening this discussion. I came to the same conclusion when I was looking into this issue a while ago. Unfortunately I don't have ideas on how to solve it properly because it's more something that needs to be addressed by the HCCM.

For us at Brella we don't use private networks anyway since our clusters can scale beyond the max 100 nodes that private networks support. But many hetzner-k3s users do use private networks as default config because it's simpler. What happened when you disabled the route controller?