What happened:
We manage the policyendpoints.networking.k8s.aws CRD ourselves outside of the aws-vpc-cni Helm chart. This is a self managed VPC CNI installation not the EKS managed add on. The chart stopped shipping this CRD as of aws-vpc-cni v1.22.4. See aws/amazon-vpc-cni-k8s#3623 and the fix in aws/amazon-vpc-cni-k8s#3625. That change assumes the CRD is always installed and owned by the EKS control plane. We have now observed that this behaves inconsistently across our clusters.
Environment A: our automation deleted the CRD due to an unrelated infrastructure as code regression that stopped tracking it. Within a few hours the CRD reappeared automatically. It was owned by field managers controller and kube apiserver, with a brand new uid and resourceVersion, so this was a genuine recreation not a stale read. This coincided with a rollout of the aws node DaemonSet meaning a VPC CNI version or config change. It did not coincide with any EKS platform version bump or Kubernetes minor version upgrade on that cluster.
Environment B: the same class of infrastructure as code regression deleted the CRD but it was not restored. Every live NetworkPolicy object had its corresponding PolicyEndpoint cascade deleted along with the CRD, and nothing recreated either the CRD or the PolicyEndpoint objects afterwards. NetworkPolicy enforcement silently stopped and traffic failed open, while kubectl get netpol continued to show a healthy deny by default posture since the NetworkPolicy objects themselves were untouched.
Both environments run self managed VPC CNI installed via Helm not the EKS managed add on, so we would expect the same recovery behavior in both. We are seeing the opposite.
What we are trying to understand:
What specifically triggers the EKS control plane network policy controller to install or reinstall the policyendpoints.networking.k8s.aws CRD when it is missing. Is it a per node or DaemonSet event such as an aws node pod restart or VPC CNI version change, as our Environment A observation suggests. Or is it tied to an EKS cluster or platform version update or some other control plane side reconciliation trigger. Or is it a periodic reconciliation loop with a fixed interval, or something else entirely.
Why would the same deletion event self heal on one cluster but not another when both are configured the same way with self managed CNI and enableNetworkPolicy set to true. Is there a precondition such as a minimum EKS platform version, minimum CNI version, or a specific aws node or network policy agent version, required for the control plane to detect and restore the CRD.
For self managed VPC CNI customers specifically, is control plane driven self healing of this CRD a supported and guaranteed behavior we can rely on, or should we assume it will never reliably recover and always vendor and manage this CRD ourselves as suggested for self managed clusters in aws/amazon-vpc-cni-k8s#3625.
If the CRD is deleted and later restored by the control plane, is PolicyEndpoint reconciliation for pre existing NetworkPolicy objects guaranteed to happen automatically afterwards, or can that require manual intervention such as an AWS support triggered reconcile in some cases.
Clarifying the actual trigger and reliability of this behavior would help us decide whether self managed CNI users can safely treat control plane installation as a source of truth, or whether we must always vendor and own this CRD ourselves going forward.
Attach logs:
Not applicable. This is a request for clarification on documented control plane behavior rather than a specific reproducible defect in the agent or controller code. Happy to provide aws-cni-support.sh output from either environment if useful for investigation.
What you expected to happen:
Consistent and documented behavior for when and how the EKS control plane installs or restores the policyendpoints.networking.k8s.aws CRD, so self managed VPC CNI customers can plan around it reliably.
How to reproduce it as minimally and precisely as possible:
On a self managed VPC CNI cluster installed via Helm not the EKS managed add on, with enableNetworkPolicy set to true and one or more live NetworkPolicy objects, delete the policyendpoints.networking.k8s.aws CRD using kubectl delete crd policyendpoints.networking.k8s.aws.
Observe whether the CRD is restored automatically, and if so what cluster or node level event coincides with its restoration.
Compare this behavior across clusters at different EKS platform versions, VPC CNI versions, and Kubernetes minor versions.
Anything else we need to know?:
We are specifically trying to distinguish a node or DaemonSet refresh from a cluster or control plane update as the trigger, since that materially changes how self managed CNI customers should design around this dependency. For example whether restarting aws node is sufficient to recover, versus needing to wait for or force a control plane side event.
Environment:
- Kubernetes version (use kubectl version): 1.34 to 1.36 observed across multiple clusters
- CNI Version: 1.22.4
- Network Policy Agent Version: 1.4.0
- OS (e.g: cat /etc/os-release): Amazon Linux 2023 EKS optimized AMI
- Kernel (e.g. uname -a): 5.10 or later EKS optimized AMI default
What happened:
We manage the policyendpoints.networking.k8s.aws CRD ourselves outside of the aws-vpc-cni Helm chart. This is a self managed VPC CNI installation not the EKS managed add on. The chart stopped shipping this CRD as of aws-vpc-cni v1.22.4. See aws/amazon-vpc-cni-k8s#3623 and the fix in aws/amazon-vpc-cni-k8s#3625. That change assumes the CRD is always installed and owned by the EKS control plane. We have now observed that this behaves inconsistently across our clusters.
Environment A: our automation deleted the CRD due to an unrelated infrastructure as code regression that stopped tracking it. Within a few hours the CRD reappeared automatically. It was owned by field managers controller and kube apiserver, with a brand new uid and resourceVersion, so this was a genuine recreation not a stale read. This coincided with a rollout of the aws node DaemonSet meaning a VPC CNI version or config change. It did not coincide with any EKS platform version bump or Kubernetes minor version upgrade on that cluster.
Environment B: the same class of infrastructure as code regression deleted the CRD but it was not restored. Every live NetworkPolicy object had its corresponding PolicyEndpoint cascade deleted along with the CRD, and nothing recreated either the CRD or the PolicyEndpoint objects afterwards. NetworkPolicy enforcement silently stopped and traffic failed open, while kubectl get netpol continued to show a healthy deny by default posture since the NetworkPolicy objects themselves were untouched.
Both environments run self managed VPC CNI installed via Helm not the EKS managed add on, so we would expect the same recovery behavior in both. We are seeing the opposite.
What we are trying to understand:
What specifically triggers the EKS control plane network policy controller to install or reinstall the policyendpoints.networking.k8s.aws CRD when it is missing. Is it a per node or DaemonSet event such as an aws node pod restart or VPC CNI version change, as our Environment A observation suggests. Or is it tied to an EKS cluster or platform version update or some other control plane side reconciliation trigger. Or is it a periodic reconciliation loop with a fixed interval, or something else entirely.
Why would the same deletion event self heal on one cluster but not another when both are configured the same way with self managed CNI and enableNetworkPolicy set to true. Is there a precondition such as a minimum EKS platform version, minimum CNI version, or a specific aws node or network policy agent version, required for the control plane to detect and restore the CRD.
For self managed VPC CNI customers specifically, is control plane driven self healing of this CRD a supported and guaranteed behavior we can rely on, or should we assume it will never reliably recover and always vendor and manage this CRD ourselves as suggested for self managed clusters in aws/amazon-vpc-cni-k8s#3625.
If the CRD is deleted and later restored by the control plane, is PolicyEndpoint reconciliation for pre existing NetworkPolicy objects guaranteed to happen automatically afterwards, or can that require manual intervention such as an AWS support triggered reconcile in some cases.
Clarifying the actual trigger and reliability of this behavior would help us decide whether self managed CNI users can safely treat control plane installation as a source of truth, or whether we must always vendor and own this CRD ourselves going forward.
Attach logs:
Not applicable. This is a request for clarification on documented control plane behavior rather than a specific reproducible defect in the agent or controller code. Happy to provide aws-cni-support.sh output from either environment if useful for investigation.
What you expected to happen:
Consistent and documented behavior for when and how the EKS control plane installs or restores the policyendpoints.networking.k8s.aws CRD, so self managed VPC CNI customers can plan around it reliably.
How to reproduce it as minimally and precisely as possible:
On a self managed VPC CNI cluster installed via Helm not the EKS managed add on, with enableNetworkPolicy set to true and one or more live NetworkPolicy objects, delete the policyendpoints.networking.k8s.aws CRD using kubectl delete crd policyendpoints.networking.k8s.aws.
Observe whether the CRD is restored automatically, and if so what cluster or node level event coincides with its restoration.
Compare this behavior across clusters at different EKS platform versions, VPC CNI versions, and Kubernetes minor versions.
Anything else we need to know?:
We are specifically trying to distinguish a node or DaemonSet refresh from a cluster or control plane update as the trigger, since that materially changes how self managed CNI customers should design around this dependency. For example whether restarting aws node is sufficient to recover, versus needing to wait for or force a control plane side event.
Environment: