Need help on deployment/setup

[ArgirisGkogkidis]

Hello everyone,

First of all, congratulations and many thanks for this amazing tool. I have started exploring kubernetes as a migration from our docker swarm cluster, since things have started getting heavy.

I am still trying to get familiar with the terminology, so apologies in advance if I mix things up, but please be direct and point me to the correct terminology/path if possible.

What I have/had on mind, what a setup like this:

So I have a bastion VM, which is also used as NAT to the cluster nodes to be able to install k3s, patch OS udpates etc. Also we did setup wireguard there for our VPN connection.

Then, the idea is to have Cloudflare to maintain the A records -> point to hetzner loadbalancer -> talk to cluster nodes and expose API's, websites etc.
At least that was the flow we did manage with docker swarm. hetzner load balancer was talking with 3 master nodes where traefik was there as reverse proxy.

Now in kubernetes setup, I understand we have 2 points:
Ingress and Egress.
So, by setting

traefik:
    enabled: true 

This means that the cluster is using internally (ingress) the traefik instead of NGINX (which I read is getting terminated (not supported) lets say in the coming months).

So, questions

1. for Egress, how do I expose my NextJS, Wordpress, xxx API to the world?
2. does it make sense to use Loadbalancer to talk to the cluster? if so does the loadbalancer need to talk to all nodes? in private network of course.

kubectl -n kube-system get svc traefik
NAME      TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)                      AGE
traefik   LoadBalancer   10.43.242.81   <pending>     80:31916/TCP,443:31077/TCP   7d20h

I would expect that this would have traefik in HA. since we have 3 nodes for control-plane. Is this expected from the initial setup and then we scale this up?

My configuration so far.

hetzner_token: *********
cluster_name: ***********
kubeconfig_path: "./kubeconfig"
k3s_version: v1.34.1+k3s1

networking:
  ssh:
    port: 22
    use_agent: false # set to true if your key has a passphrase
    public_key_path: "~/.ssh/id_ed25519.pub"
    private_key_path: "~/.ssh/id_ed25519"
  allowed_networks:
    ssh:
      - **********************************************************/32
      - 10.0.0.0/16
      - 10.8.0.0/24                # WireGuard VPN cidr if used
    api:                            # k8s API (6443) reachable only privately
      - **********************************************************/32
      - 10.0.0.0/16
      - 10.8.0.0/24
    custom_firewall_rules:
      - description: "Cilium VXLAN overlay"
        direction: in
        protocol: udp
        port: 8472
        source_ips: [ "10.0.0.0/16" ]

  public_network:
    ipv4: false
    ipv6: false
    # hetzner_ips_query_server_url: https://.. # for large clusters, see https://github.com/vitobotta/hetzner-k3s/blob/main/docs/Recommendations.md
    # use_local_firewall: false # for large clusters, see https://github.com/vitobotta/hetzner-k3s/blob/main/docs/Recommendations.md
  private_network:
    enabled: true
    subnet: 10.0.0.0/16
    existing_network_name: "private-network"
  cni:
    enabled: true
    encryption: false
    mode: cilium

datastore:
  mode: etcd # etcd (default) or external
  # external_datastore_endpoint: postgres://....
  etcd:
      snapshot_retention: 24
      snapshot_schedule_cron: "0 * * * *"

      # S3 snapshot configuration (optional)
      s3_enabled: true
      s3_endpoint: "****************.com" # Can also be set with ETCD_S3_ENDPOINT environment variable
      s3_region: "eu-central-1" # Can also be set with ETCD_S3_REGION environment variable
      s3_bucket: "k3s-cluster" # Can also be set with ETCD_S3_BUCKET environment variable
      s3_access_key: "*********" # Can also be set with ETCD_S3_ACCESS_KEY environment variable
      s3_secret_key: "***********" # Can also be set with ETCD_S3_SECRET_KEY environment variable
      s3_folder: "****************-cluster"
      s3_force_path_style: false

schedule_workloads_on_masters: false

image: debian-12 # optional: default is ubuntu-24.04
autoscaling_image: debian-12 

masters_pool:
  instance_type: cx23
  instance_count: 3 
  locations:
    - fsn1
    - hel1
    - nbg1

worker_node_pools:
- name: small-scalled
  instance_type: cx33
  location: fsn1
  autoscaling:
    enabled: true
    min_instances: 1
    max_instances: 4

- name: medium-autoscaled
  instance_type: cx43
  location: fsn1
  autoscaling:
    enabled: true
    min_instances: 0
    max_instances: 3

cluster_autoscaler:
  scan_interval: "10s"                        # How often cluster is reevaluated for scale up or down
  scale_down_delay_after_add: "10m"           # How long after scale up that scale down evaluation resumes
  scale_down_delay_after_delete: "10s"        # How long after node deletion that scale down evaluation resumes
  scale_down_delay_after_failure: "3m"        # How long after scale down failure that scale down evaluation resumes
  max_node_provision_time: "15m"              # Maximum time CA waits for node to be provisioned

embedded_registry_mirror:
  enabled: false

addons:
  csi_driver:
    enabled: false   # Hetzner CSI driver (default true). Set to false to skip installation.
  traefik:
    enabled: true  # built-in Traefik ingress controller. Disabled by default.
  servicelb:
    enabled: false  # built-in ServiceLB. Disabled by default.
  metrics_server:
    enabled: true  # Kubernetes metrics-server addon. Disabled by default.
  cloud_controller_manager:
    enabled: true   # Hetzner Cloud Controller Manager (default true). Disabling stops automatic LB provisioning for Service objects.
  cluster_autoscaler:
    enabled: true   # Cluster Autoscaler addon (default true). Set to false to omit autoscaling.

protect_against_deletion: true
create_load_balancer_for_the_kubernetes_api: false
k3s_upgrade_concurrency: 1

additional_pre_k3s_commands:
- apt update
- apt upgrade -y
- echo "auto enp7s0" >> /etc/network/interfaces
- echo "iface enp7s0 inet dhcp" >> /etc/network/interfaces
- echo "    post-up ip route add default via 10.0.0.1"  >> /etc/network/interfaces
- echo "dns-nameservers 8.8.8.8 1.1.1.1" >> /etc/network/interfaces
- echo "[Resolve]" > /etc/systemd/resolved.conf
- echo "DNS=185.12.64.2 185.12.64.1" >> /etc/systemd/resolved.conf
- echo "FallbackDNS=8.8.8.8" >> /etc/systemd/resolved.conf
- ifdown enp7s0 && ifup enp7s0
- systemctl start systemd-resolved
- systemctl enable systemd-resolved

additional_post_k3s_commands:
  - 'export DEBIAN_FRONTEND=noninteractive; apt-get update -y && apt-get install -y fail2ban ca-certificates curl gnupg lsb-release unattended-upgrades systemd-resolved systemd-timesyncd'
  - 'dpkg-reconfigure --priority=low unattended-upgrades'
  - 'sed -i "s#^//\\?Unattended-Upgrade::Automatic-Reboot .*#Unattended-Upgrade::Automatic-Reboot \\"true\\";#" /etc/apt/apt.conf.d/50unattended-upgrades'
  - 'grep -q "Unattended-Upgrade::Automatic-Reboot-Time" /etc/apt/apt.conf.d/50unattended-upgrades || echo "Unattended-Upgrade::Automatic-Reboot-Time \\"04:00\\";" >> /etc/apt/apt.conf.d/50unattended-upgrades'
  - 'systemctl enable --now unattended-upgrades fail2ban systemd-resolved systemd-timesyncd'

[vitobotta]

Hi,

• Load balancers: don't confuse the load balancer created by hetzner-k3s with load balancers for your workloads. If you turn on the load balancer in the hetzner-k3s configuration, that load balancer is dedicated to access the Kubernetes API from clients when the cluster is in HA configuration with multiple masters. This load balancer is not intended for use with anything other than the Kubernetes API. I am clarifying this just in case since sometimes people who are starting out get confused about this load balancer. Also, one important note in the case you decide to enable this load balancer in the configuration. While using a load balancer to access the Kubernetes API is convenient (since it load balances requests to the masters automatically), there is a security caveat: load balancers are not yet supported in the Hetzner Cloud firewalls, so even if you specify some allowed networks for the Kubernetes API in the hetzner-k3s configuration, that load balancer will still be exposed to the public Internet, i.e. the Kubernetes API will be exposed to the public Internet. For some people that's fine, but IMO for critical and production clusters I wouldn't do that until Hetzner adds support for the load balancers in the firewalls. Until them, I'd keep the load balancer turned off in the configuration. In that case, hetzner-k3s will create a kubeconfig file that includes one context for each master in the control plane, so the only downside is that you need to switch context manually if the current context points to a master that is having trouble.

• Ingress: correct. If you turn on Traefik in the configuration, the tool will enable the version of Traefik that is built in in k3s, so you won't have to install a separate ingress controller. However, I recommend you look into the newer API gateways instead, and install one yourself since you would have more control on its configuration compared to k3s' Traefik. That said, I haven't used k3s' built in Traefik myself so I am not too familiar with it.

• Exposing apps to the Internet: you have 3 options.

  • 1. you install an ingress controller (a still supported one) or an API gateway with the service configured to use a network load balancer, then you configure an ingress (or equivalent resource for the API gateway) for each web application etc. That way all the applications will share the same load balancer. Note that if you use k3s' built in Traefik controller I don't know if you can set up a network load balancer. I think it will use a "fake" load balancer using the IPs of the cluster nodes or something like that. So do your research before going that router.
  • 2. you have a separate load balancer for each application. It's more expensive, and it will give you less flexibility compared to using an ingress controller or API gateway.
  • 3. Cloudflare tunnels. This can increase security by establishing a direct tunnel between your applications and the Internet, and is cheaper since you don't need load balancers with it, but it comes with some limitations.

Hopefully others with a setup similar to yours can offer more advice.

[ArgirisGkogkidis]

Hi @vitobotta, much appreciated the prompt response.

So in the first case, the load balancer is to "load balance" the kubectl commands, let's say towards the cluster? Otherwise, I will need to manually change the context to which master node I am issuing a command? Then I could use my bastion as a LB towards the master nodes? just a dummy idea here.

Ingress perfectly clear I will check the newer API gateway and see what fits our purpose for now at least.

Exposing to the internet, Cloudflare tunnels will bind me to always use CF right? not all the domains we manage are through CF unfortunatelly, and I can not avoid that 100%. Especially with the outage last week, we are trying to find way to mitigate this in the future.
option B, well yes but no. Ideally less moving components since we are small team.
I will explore option A and I will report back how we did.

And again apologies if I mix up things or use wrong terms, please correct me.

[vitobotta]

Hi @vitobotta, much appreciated the prompt response.

So in the first case, the load balancer is to "load balance" the kubectl commands, let's say towards the cluster? Otherwise, I will need to manually change the context to which master node I am issuing a command? Then I could use my bastion as a LB towards the master nodes? just a dummy idea here.

Yep. At work we simply run commands with a wrapper that automatically selects a random but working master context when starting a kubectl session or with other client.

Ingress perfectly clear I will check the newer API gateway and see what fits our purpose for now at least.

Exposing to the internet, Cloudflare tunnels will bind me to always use CF right? not all the domains we manage are through CF unfortunatelly, and I can not avoid that 100%.

Correct. I mentioned it just to give you another option since you mentioned Cloudflare in your original post. :)

Especially with the outage last week, we are trying to find way to mitigate this in the future. option B, well yes but no. Ideally less moving components since we are small team. I will explore option A and I will report back how we did.

And again apologies if I mix up things or use wrong terms, please correct me.

[ArgirisGkogkidis]

just an update so far, I will try to compile everything later and perhaps a PR to update the documentation.

For now at least we will try traefik to maintain (in theory at least) an easy migration for our setup.
So, after cluster setup, in order to expose traefik to loadbalancer for ingress, we need to run this:

kubectl -n kube-system patch svc traefik \
  -p '{
    "metadata": {
      "annotations": {
        "load-balancer.hetzner.cloud/location": "fsn1", # choose your location
        "load-balancer.hetzner.cloud/use-private-ip": "true",
        "load-balancer.hetzner.cloud/name": "traefik-lb",
        "load-balancer.hetzner.cloud/http-redirect-https": "false"
      }
    }
  }'

and to enable traefik features as per k3s documentaiton, we need to create a new file.
So we did create traefik-config.yaml

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: traefik          # must match HelmChart name
  namespace: kube-system # must be kube-system
spec:
  valuesContent: |-
    # Extra static arguments passed to Traefik
    additionalArguments:
      # Redirect HTTP -> HTTPS
      - "--entrypoints.web.http.redirections.entrypoint.to=websecure"
      - "--entrypoints.web.http.redirections.entrypoint.scheme=https"
      - "--entrypoints.web.http.redirections.entrypoint.permanent=true"

      # ACME (Let's Encrypt) HTTP-01 on web (port 80)
      - "--certificatesresolvers.myresolver.acme.httpchallenge=true"
      - "--certificatesresolvers.myresolver.acme.httpchallenge.entrypoint=web"
      - "--certificatesresolvers.myresolver.acme.email=info@company.com"
      - "--certificatesresolvers.myresolver.acme.storage=/data/acme.json"

    # Single replica for now – simpler, works with local-path PVC
    deployment:
      replicas: 1

    # Persist /data (acme.json) using default local-path provisioner
    persistence:
      enabled: true
      path: /data
      size: 1Gi
      # storageClass: ""   # leave empty to use default 'local-path'

    ports:
      websecure:
        tls:
          enabled: true

In our case we want to deploy longhorn and share storage between nodes (not to use hcloud volumes), we need to include the following in additional_pre_k3s_commands:

apt update
apt install -y open-iscsi nfs-common
systemctl enable --now iscsid

and on our cluster-config, we have this:

addons:
  csi_driver:
    enabled: false   # Hetzner CSI driver (default true). Set to false to skip installation.

Questions:

1. are we on the right path?
2. If we do this on existing configuration, and then we need to use create or upgrade?

hetzner-k3s create --config cluster_config.yaml

hetzner-k3s upgrade --config cluster_config.yaml

or we need to delete cluster and restart this?

3. if someone has tested, does it make sense to use longhorn or try the hcloud volumes? thus we enable the csi_driver? if we are to use hcloud volumes, by default as is the configuration right now, replicas are happening how? in documentation https://vitobotta.github.io/hetzner-k3s/Storage/ it is mentioned "Replicated: Based on Ceph, ensuring data is replicated across multiple disks" but on the create volume interface it states the volume is bound to the location of the server. currently our workers are on the same server location, does this mean, if a node goes down that volume will be mounted automatically on the other node and pod will transition to that?

Edit: there is a reply here about the replication of hcloud volumes noticed that late #360