CronJob pods with names truncated to 63 chars share one podIdentifier across all runs; overlapping runs lose return traffic (ingress BPF_DROP) until the shared program is unpinned

[ngnix]

What happened:

GetPodIdentifier (pkg/utils/utils.go) derives the pod identifier by dropping the last --separated token of the pod name, assuming the name always ends with -<random-suffix>. For CronJob pods the name is <cronjob>-<8-digit-minutes>-<5-char-random>. When the CronJob name is longer than 48 characters, the API server truncates the generated prefix to keep the pod name within 63 characters, and the - between the Job timestamp and the random suffix is removed. Timestamp and random suffix fuse into one token, so every run of that CronJob maps to the same podIdentifier. Shorter CronJobs keep a unique identifier per run.

cronjob name (52 chars): example-app-backend-worker-queue-notification-sender
pod name (63 chars):     example-app-backend-worker-queue-notification-sender-29808n7p4d
podIdentifier:           example-app-backend-worker-queue-notification-sender-team-a            <- shared by ALL runs

cronjob name (48 chars): example-app-backend-worker-queue-cache-evictions
pod name (63 chars):     example-app-backend-worker-queue-cache-evictions-29808480-xkxxh
podIdentifier:           example-app-backend-worker-queue-cache-evictions-29808480-team-a       <- unique per run

(Identifier format shown for v1.3.x, <prefix>-<namespace>; v1.4.x uses @ as separator, prefix derivation is unchanged.)

aws-eks-na-cli ebpf loaded-ebpfdata on a node confirms it: identifiers without the Job timestamp for CronJobs whose names exceed 48 chars, timestamped identifiers for the rest.

Pod Identifier : example-app-backend-worker-queue-notification-sender-team-a          Direction : egress
Pod Identifier : example-app-backend-worker-queue-archived-keys-remover-team-a        Direction : egress
Pod Identifier : example-app-backend-worker-queue-cache-evictions-29809028-team-a     Direction : egress
Pod Identifier : example-app-backend-worker-queue-new-language-29809028-team-a        Direction : egress

In normal operation this goes unnoticed: a run finishes and its DeletePodNp unpins the program and maps before the next run's EnforceNpToPod arrives, so each run still sees "No map instance found" and loads a fresh program (agent log on one node: 1632 EnforceNpToPod, 1617 DeletePodNp, only 2 "Found an existing instance").

The failure starts as soon as two pods of the same CronJob overlap (one pod in CrashLoopBackOff with restartPolicy: OnFailure, next run scheduled 2 minutes later). From that moment every new pod of that CronJob loses return traffic:

• Pods resolve DNS via node-local-dns on the kube-dns ClusterIP. The UDP request leaves the pod and is answered (node-local-dns metrics: requests +5x, all answered NOERROR/NXDOMAIN, 0 SERVFAIL). The reply never reaches the pod. glibc times out after 20 s: could not translate host name "db-pooler-rw" to address: Temporary failure in name resolution.
• Same node, same minute: 11 other CronJobs in the same namespace (same image, same DB host, some sleeping longer before connecting) worked fine. Pod IPs of the failing pods differed on every run and came from the same prefixes as the healthy pods.
• The failure persisted across container restarts inside the same pod (6 restarts, 20 s each) and across ~100 consecutive pods for 3 h 15 min. It stopped by itself once no overlapping pods of that CronJob remained.
• Agent log for the shared identifier shows the program and maps being deleted on each run's delete:

Deleting: Program: /sys/fs/bpf/globals/aws/programs/example-app-backend-worker-queue-notification-sender-team-a_handle_egress Map: /sys/fs/bpf/globals/aws/maps/example-app-backend-worker-queue-notification-sender-team-a_egress_map Map: /sys/fs/bpf/globals/aws/maps/example-app-backend-worker-queue-notification-sender-team-a_egress_pod_state_map
Updating pod_state map to default allow/default deny for podIdentifier: example-app-backend-worker-queue-notification-sender-team-a, state: 1

The observable symptom (egress passes, ingress reply dropped) matches tc.v4ingress.bpf.c: a missing pod_state_map entry returns BPF_DROP with ERROR_TIER. With a shared identifier, EnforceNpToPod evaluates IsFirstPodInPodIdentifier outside the per-identifier lock and then either reuses the existing program ("Found an existing instance") or loads a new one under the lock, while a concurrent DeletePodNp for a sibling pod of the same identifier can unpin the program and maps the new pod is about to share. Once one pod of the identifier is in this state, every later pod attaches to the same program (context exists, ProgFD != 0), so the whole CronJob stays broken until the last pod of the identifier is deleted and the program is finally unpinned.

No NetworkPolicy selects these pods. Enforcement is in standard mode. Probes are attached to all pods (expected since #361).

Attach logs

Agent logs from the affected node are no longer available (spot node was terminated). Logs from a healthy node showing the shared identifiers and the delete pattern can be sent to k8s-awscni-triage@amazon.com on request.

What you expected to happen:

Pods of different Job runs must never share an eBPF program, policy maps or pod_state maps. The identifier should come from something unique per controller instance, e.g. ownerReferences[0].uid (Job UID) for Job-owned pods, or the full pod name when the name was truncated. IsFirstPodInPodIdentifier should be evaluated under the same per-identifier lock as attach and delete.

How to reproduce it (as minimally and precisely as possible):

1. Cluster with VPC CNI network policy enabled, NETWORK_POLICY_ENFORCING_MODE=standard, no NetworkPolicy in the namespace.
2. Create a CronJob whose name is longer than 48 characters, schedule */2 * * * *, concurrencyPolicy: Allow, restartPolicy: OnFailure, successfulJobsHistoryLimit: 3, failedJobsHistoryLimit: 1. Container command: sleep 60 && getent hosts kubernetes.default.svc.cluster.local.
3. Verify with aws-eks-na-cli ebpf loaded-ebpfdata on the node that the identifier has no Job timestamp (<cronjob-name>-<namespace>).
4. Make two runs overlap: e.g. make one run fail (exit 1 after the sleep) so it stays in CrashLoopBackOff while the next run starts, or set the schedule to every minute with sleep 90.
5. Observe: the new run's DNS lookup times out ("Temporary failure in name resolution") while a control CronJob with a name of 48 characters or less and identical spec keeps working on the same node. The failure persists for all following runs until no pod of the long-named CronJob is running.

Anything else we need to know?:

• Related: Network policy agent does not support Standalone Pods #327 (pods sharing a name prefix are treated as one deployment), Recreated same-named Job pods inherit stale NetworkPolicy enforcement via podIdentifier after pod-template labels change to be exempt (standard mode) #616 (recreated same-named Job pods inherit stale state via podIdentifier), Roll back orphaned eBPF programs on partial attach failure (AttacheBPFProbes) #621 / Question: is there any garbage collection for orphaned pinned eBPF maps/programs on bpffs? #618 (Job identifiers and pin lifecycle), Fix race condition with attach and delete concurrency #452 (attach/delete race for short-lived pods), Policies with names >=58 characters collapse to the same derived key #665 (same last-hyphen truncation problem for PolicyEndpoint names).
• GetPodIdentifier is unchanged between v1.3.4 and v1.4.2 (only the namespace separator changed in fix(utils): make GetPodIdentifier injective to prevent cross-namespac… #576), so upgrading does not fix this.
• Workaround we apply: keep CronJob names at or below 48 characters and set concurrencyPolicy: Forbid.

Environment:

• Kubernetes version (use kubectl version): Server v1.35.6-eks-bca9cf6, kubelet v1.35.7-eks-cb19647
• CNI Version: amazon-k8s-cni v1.21.1-eksbuild.8 (managed addon), AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG=true, ENABLE_PREFIX_DELEGATION=true
• Network Policy Agent Version: v1.3.4-eksbuild.1, NETWORK_POLICY_ENFORCING_MODE=standard, --enable-policy-event-logs=true
• OS (e.g: cat /etc/os-release): Amazon Linux 2023.12.20260817
• Kernel (e.g. uname -a): 6.12.100-125.179.amzn2023.x86_64, containerd 2.2.5