Race on same-node pod replacement leaves the agent updating an egress map no attached program reads (pods blocked or silently unenforced)

[razorsk8jz]

What happened:

A DaemonSet pod selected by an egress NetworkPolicy was deleted and recreated on the same node (a scheduled daily restart). From that point every replacement pod of that DaemonSet on that node came up in one of two broken states, depending on ordering:

• Blocked: zero egress from the first packet. The attached egress program enforced against an empty map. Adding rules to the policy, removing the pod from the policy, or deleting the pod did not change anything.
• Silently unenforced: the pod ran default-allow for a full day even though a policy selected it. Nothing in the cluster indicated this.

Root cause, from the agent log on the node and bpftool against the pinned objects:

1. All eBPF state is keyed by podIdentifier (pod name minus its last -segment, plus namespace). Programs and maps are pinned under /sys/fs/bpf/globals/aws/{programs,maps}/<identifier>_* and shared by every pod with that identifier on the node.
2. When the old pod was deleted, the PolicyEndpoint reconcile and the CNI DEL for that pod interleaved:
   • The CNI DEL removed the identifier's programs, maps and context.
   • A PolicyEndpoint reconcile that still listed the terminating pod then loaded a new program set for the already-dead pod (handle_egress prog A, egress map M1) and cached M1 as the identifier's in-memory egress map. Log (paraphrased): didn't find egress in-mem map ... creating new In memory map ... MapID:<M1>.
   • ~600 ms later the CNI ADD for the replacement pod found no egress program for the identifier and loaded a second set (prog B, egress map M2), pinned it over prog A's pin, and stored prog B's FD as the identifier's program. Prog A survived only through a leaked FD.
3. Every later pod with that identifier attaches prog B (Found an existing instance), which reads M2. Every rule update from the reconciler goes to M1. Nothing ever re-derives the in-memory cache from the pinned objects, so the divergence persists until the agent process restarts or the node is replaced.
4. Which of the two outcomes a pod gets is decided by egress_pod_state_map key 0, which is written through the live program FD and therefore does reach the attached program's map:
   • CNI ADD path, when the PolicyEndpoint selector is momentarily empty: No active policies present -> key0 = 1 (default-allow) -> unenforced.
   • Reconcile path afterwards: Active policies available ... No Map update required -> key0 = 0 -> enforce against the empty M2 -> blocked.

Only one node was affected in a fleet of six; only the one DaemonSet identifier on that node was affected. Pods of other names on the same node were fine. A curl pod given a name with the same prefix, default service account and no labels was blocked immediately, which is what pointed at the identifier-keyed shared state.

Attach logs

Node log excerpt (network-policy-agent.log, ebpf-sdk.log) covering the replacement window, plus bpftool prog show/map show/map dump pinned output, will be sent to k8s-awscni-triage@amazon.com referencing this issue. Note this is EKS Auto Mode (Bottlerocket), so aws-cni-support.sh is not available; the data was collected from a kubectl debug node --profile=sysadmin pod.

What you expected to happen:

• One program set per podIdentifier on a node at any time, or an explicit rebuild of the in-memory cache from the pinned objects when a load finds an existing pin.
• The reconciler to write rules to the map the attached program actually reads, or to detect and repair the mismatch (compare the cached map id with the map_ids of the pinned/attached handle_egress program on each update).
• A pod selected by a policy never to run default-allow because its state map was written during a transient empty-selector window.

How to reproduce it (as minimally and precisely as possible):

This is a race; in our environment it hit roughly 1 in 40–50 same-node replacements.

1. EKS Auto Mode cluster with the network policy controller enabled (amazon-vpc-cni ConfigMap enable-network-policy-controller: "true"), NodeClass networkPolicy: DefaultAllow.
2. A DaemonSet whose pods are selected by an egress NetworkPolicy allowing a few TCP ports (in our case 15008, 15020, 53).
3. Repeatedly delete the DaemonSet pod on one node so the replacement lands on the same node (a CronJob doing kubectl rollout restart daemonset/... daily reproduces the timing). The window is the interval between the CNI DEL of the old pod and the CNI ADD of the new one, when the PolicyEndpoint update for the deletion may still list the old pod.
4. After a replacement, on the node compare:
   • bpftool prog show pinned /sys/fs/bpf/globals/aws/programs/<identifier>_handle_egress -> map_ids
   • the map id the agent logs as the identifier's in-memory egress map
     If they differ, the state is corrupted. bpftool map dump pinned .../<identifier>_egress_map on the attached program's map shows zero entries while the policy has rules; <identifier>_egress_pod_state_map key 0 tells you whether that pod is blocked (0) or unenforced (1).

Anything else we need to know?:

• Agent restart on the node clears it (the agent re-reads the pinned state on start). Deleting pods, waiting, or toggling policy membership does not; the pins are never re-derived.
• The unenforced outcome does not appear to be reported anywhere; it is the more serious of the two because nothing alerts on it.
• Detection that worked for us: with Istio ambient, ztunnel logs connection timed out, maybe a NetworkPolicy is blocking HBONE port 15008 and increments istio_tcp_connections_closed_total{response_flags="NETWORK_POLICY"} on the affected node only. Switching the NodeClass to networkPolicy: DefaultDeny converts the unenforced outcome into the blocked one, which is at least observable.
• Suggested fixes: (a) serialize CNI DEL/ADD and PolicyEndpoint reconcile per identifier, (b) on Found an existing instance, rebuild the in-memory map references from the pinned program's map_ids rather than trusting the cache, (c) a periodic consistency check of cached map ids against attached programs, (d) never write pod_state = default-allow for an identifier that currently has a policy in the PolicyEndpoint set.

Environment:

• Kubernetes version (use kubectl version): v1.36.2-eks
• CNI Version: EKS Auto Mode built-in CNI (not the vpc-cni addon)
• Network Policy Agent Version: 1.0.1037.0-1 (ebpf-sdk v1.0.16), as shipped in the Auto Mode node image
• OS (e.g: cat /etc/os-release): Bottlerocket (EKS Auto, Standard) 2026.8.26, aws-k8s-1.36-standard
• Kernel (e.g. uname -a): 6.18.39 (Bottlerocket)

Related issues in this repo:

• When aws-node pod is recreated it will load and pin a new eBPF program with new maps, but the tc filters will sometimes still reference the old program #514 — new eBPF program pinned while tc filters still reference the old one (same class: attached program vs pinned/cached state diverge; different trigger — agent restart rather than pod replacement)
• Recreated same-named Job pods inherit stale NetworkPolicy enforcement via podIdentifier after pod-template labels change to be exempt (standard mode) #616 — recreated same-named pods inherit stale enforcement through the shared podIdentifier
• CronJob pods with names truncated to 63 chars share one podIdentifier across all runs; overlapping runs lose return traffic (ingress BPF_DROP) until the shared program is unpinned #671 — CronJob pods sharing one podIdentifier across runs
• Pod eBPF Map Differs from Policy Endpoint #206 — pod eBPF map differs from PolicyEndpoint, cleared by agent restart (closed)
• Network traffic sporadically denied despite valid network policies #307 — traffic sporadically denied despite valid network policies (closed)
• Race condition can leads to egress connectivity #83 — race condition leading to loss of egress connectivity (closed)