Allow configuring the Kubernetes API server endpoint to avoid kube-proxy startup dependency

[tn20000]

What happened:

On an EKS cluster where kube-proxy's initial iptables sync takes longer than usual, a new node can become Ready and start workloads before the Kubernetes Service ClusterIP is usable on that node. The network policy agent builds its Kubernetes client with rest.InClusterConfig(), so it initially targets that Service ClusterIP.

The resulting startup sequence is:

1. the network policy agent starts before kube-proxy's initial sync completes;
2. API requests to the Kubernetes Service IP time out;
3. the PolicyEndpoint informer cache does not sync before its deadline;
4. the manager exits and the agent restarts; and
5. a later attempt succeeds after kube-proxy finishes programming Service rules.

This has been observed consistently across new-node launches in the affected cluster.

Attach logs

Representative, anonymized messages:

Get "https://<kubernetes-service-ip>:443/apis/networking.k8s.aws/v1alpha1/policyendpoints?...": dial tcp <kubernetes-service-ip>:443: i/o timeout
problem running manager: failed to wait for policyendpoint caches to sync

The network policy agent's RuntimeConfig already has an APIServer field, but it is not bound to a CLI flag and BuildRestConfig does not use it.

What you expected to happen:

The agent should optionally accept a direct Kubernetes API server URL, while continuing to use the normal in-cluster service-account token and CA configuration. When configured with the EKS API endpoint, informer cache sync can proceed before kube-proxy has programmed the Kubernetes Service ClusterIP, avoiding the startup restart loop.

A narrow implementation would:

1. expose the existing field as an optional --apiserver flag;
2. construct the REST config as it does today (in-cluster or --kubeconfig); and
3. override only rest.Config.Host when --apiserver is set.

The default remains unchanged. This does not bypass informer cache synchronization; it only removes kube-proxy's Service VIP from the agent's bootstrap dependency chain. The deployment layer would still be responsible for supplying the endpoint.

How to reproduce it (as minimally and precisely as possible):

1. Delay kube-proxy's first iptables sync on a new node.
2. Start aws-node with network policy enabled on that node.
3. Allow the node to become Ready before kube-proxy finishes its initial sync.
4. Observe the network policy agent time out connecting to the Kubernetes Service ClusterIP and exit after its PolicyEndpoint cache-sync deadline.
5. Observe a subsequent restart succeed only after kube-proxy completes its first sync.

Anything else we need to know?:

The EKS API endpoint is routable directly from the node before kube-proxy is ready. Dynamically calling DescribeCluster from the agent is not necessary; the endpoint can be injected by the deployment/add-on layer. The same unused APIServer field and in-cluster-only behavior are present in released and current source.

Detailed timestamps and environment versions can be shared privately if useful.

Environment:

• Kubernetes version (use kubectl version): EKS (details available privately)
• CNI Version: details available privately
• Network Policy Agent Version: reproduced on a released version; current source has the same configuration gap
• OS: Amazon Linux
• Kernel: details available privately

[cdirubbio]

Hey! Curious if you can provide some details on the Kube proxy setup you have currently. Maybe we can solve that sync timing issue for you first.

Can you provide:

• OS (AL2,AL2023,Bottlerocket (?))
• iptables backend in-use
• Which Kube Proxy mode are you using?
• Kubernetes version

[tn20000]

Thanks! The affected nodes are:

• OS: Amazon Linux 2023 (EKS-optimized AMI)
• iptables backend: nft (iptables-nft)
• kube-proxy mode: iptables (not kube-proxy native nftables mode)
• Kubernetes version: EKS 1.33

So this is the iptables proxier running through the nft backend. If there are specific kube-proxy startup logs or metrics that would help narrow the first-sync delay, I can share sanitized output.

[cdirubbio]

Can you send the logs kube-proxy and NPA logs to k8s-awscni-triage@amazon.com?

Or if youd rather, you can just provide the first sync timing. If your scale is high enough, it might be worth considering a move to nftables mode (with some caveats https://docs.aws.amazon.com/eks/latest/best-practices/nftables.html)