Mac M1 Kubernetes - port flapper

[janhaesen]

Description

When using a profile per the attached details where Kubernetes is provided along as an argument I have some port fluttering making it impossible to use Kubernetes.

This might perhaps be related to: #1339 #1279

It is cyling through the following outputs:

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
Error from server (ServiceUnavailable): apiserver not ready

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
The connection to the server 192.168.64.2:58699 was refused - did you specify the right host or port?

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
Error from server (ServiceUnavailable): starting

To sometimes giving back a response. When doing an netcat on the host and port:

% while true; do nc -z 192.168.64.2 58699 && echo OK || echo FAIL; sleep 1; done
FAIL
Connection to 192.168.64.2 port 58699 [tcp/*] succeeded!
OK
Connection to 192.168.64.2 port 58699 [tcp/*] succeeded!
OK
Connection to 192.168.64.2 port 58699 [tcp/*] succeeded!
OK
FAIL
FAIL
FAIL
Connection to 192.168.64.2 port 58699 [tcp/*] succeeded!
OK
Connection to 192.168.64.2 port 58699 [tcp/*] succeeded!
OK
Connection to 192.168.64.2 port 58699 [tcp/*] succeeded!
OK

you can tell it sometimes is fine, but (though not visible from this snippet) more often than not it fails.

I've tried some several suggested solutions:

1. Setting the server to localhost or 127.0.0.1
2. Adding the --network-address argument
3. Using different virtualizations (with and without the --vz argument)

However none of it seems to make it stable in such a way that it can be used. Based of earlier issues it also seems as though it's primarily an issue on M1, I would find that strange, but that's coming up from the issues a bit.

Version

colima version 0.10.3
git commit: 00f6c29

runtime: docker
arch: aarch64
client: v29.5.3
server: v29.5.2
limactl version 2.1.2
qemu-img version 11.0.1
Copyright (c) 2003-2026 Fabrice Bellard and the QEMU Project developers

Operating System

• macOS Intel <= 13 (Ventura)
• macOS Intel >= 14 (Sonoma)
• Apple Silicon <= 13 (Ventura)
• Apple Silicon >= 14 (Sonoma)
• Linux

Output of colima status

INFO[0000] colima [profile=arm] is running using macOS Virtualization.Framework
INFO[0000] arch: aarch64
INFO[0000] runtime: docker
INFO[0000] mountType: virtiofs
INFO[0000] address: 192.168.64.2
INFO[0000] docker socket: unix:///Users/REDACTED/.colima/arm/docker.sock
INFO[0000] containerd socket: unix:///Users/REDACTED/.colima/arm/containerd.sock
INFO[0000] kubernetes: enabled

Reproduction Steps

1. This is where it's tricky, because a colleague with similar setup has no issues, only difference is the chip (M2)

Expected behaviour

The k8s setup is available stable and each command (kubectl cluster-info) and netcat returns the same response.

Additional context

No response

[janhaesen]

I was able to find that it becomes more stable if the engine used is containerd instead of docker. So now I realised a setup of:

1. Colima profile for x86 architecture
2. Colima profile for arm architecture (no k8s), docker engine for building
3. Colima profile for arm architecture, with Kubernetes and containerd engine
   In order to facilitate the image being available to the latter I have incorporated a docker registry from which can be pulled by the 3rd profile and pushed by the 2nd.

This also provides a way to keep base images available locally even after a clean up (not nuke from orbit).

[juliantibble]

I had this issue yesterday.

Symptoms

• I couldn't build docker images because the builds couldn't download dependencies.
• kubectl get ns from outside the VM sometimes succeeded, sometimes failed with connection refused, and sometimes failed with the Kubernetes error ServiceUnavailable
• logging into the VM with colima ssh and trying to curl a website (e.g. https://google.com) hung

Quick fix:

warning: this deletes data! - it deletes docker volumes, images, containers, and anything else in the VM

colima delete --data

Investigation

Before I ran colima delete --data I did some debugging.

The reason the kubectl get ns errors were cycling is that the k3s service was in a restart loop, so:

• if the service was temporarily up, kubectl get ns succeded
• if it was not running or hadn't started listening, kubectl get ns got connection refused
• if it was restarting, kubectl get ns got ServiceUnavailable

The reason the k3s components were restarting is that controller-manager tried to read ConfigMap kube-system/extension-apiserver-authentication but it couldn't access the Kubernetes API server, so it exited. Then k3s logged Shutdown request received: controller-manager exited.

The reason controller-manager couldn't access the API server is that it was trying to access it using the wrong port.

These three files were all generated with port 55544:

• /var/lib/rancher/k3s/server/cred/controller.kubeconfig
• /var/lib/rancher/k3s/server/cred/scheduler.kubeconfig
• /var/lib/rancher/k3s/server/cred/admin.kubeconfig

Whereas they should have been attempting to access port 50247.

As those kubeconfig files are on the data disk, they do not get deleted with colima delete. So once the VM got in this state, no amount of colima stop, colima start, colima delete, colima start would fix it until I did colima delete --data.

Networking

What was really weird was that eth0 in the VM was also not working (e.g. logging into the VM with colima ssh and trying to runcurl hung)

If I stopped and started the VM with --network-address then that adds an extra interface called col0, which does work (but eth0 still does not work). So if you also add the option --network-preferred-route then col0 is preferred over eth0 and the outbound networking from the VM appears to work (but eth0 is still broken — if you force curl to use it with the --interface flag then the requests still hang).

What was also very weird is that this eth0 issue also completely cleared up when I ran colima delete --disk and then recreated the VM. How is that possible?

Additional info

I use colima in --kubernetes mode. I usually start it with:

colima start --kubernetes --cpu 4 --memory 12