Established connections are not re-evaluated in strict mode when cluster network policies are removed

[Pavani-Panakanti]

Problem Description

In strict mode, when a pod loses its last ClusterNetworkPolicy and state flips from policies_applied to default deny, already-established connections keep passing traffic indefinitely. New connections are correctly denied. The cause is GET_CT_VAL in the eBPF datapath, which does not handle the DEFAULT_DENY pod state that strict mode uses.

How re-evaluation works

Each pod identifier has a pod-state map with two entries: entry 0 for NetworkPolicy state (POLICIES_APPLIED, or DEFAULT_ALLOW in standard mode / DEFAULT_DENY in strict mode when no NetworkPolicy applies), and entry 1 for ClusterNetworkPolicy state (POLICIES_APPLIED or DEFAULT_ALLOW).

The datapath evaluates policy once per connection and caches the verdict in aws_conntrack_map, stamped with a byte from GET_CT_VAL(entry0, entry1). Later packets compare the stored byte to the current one. Equal means reuse the cached verdict; different means re-run evaluateFlow. The byte changing is the only trigger for re-evaluating an established connection.

What happens

GET_CT_VAL tests its first argument only against DEFAULT_ALLOW and POLICIES_APPLIED. DEFAULT_DENY matches no branch and falls through to the default value, 7.

  ┌───────────────┬──────────────────┬──────┐
  │    entry 0    │     entry 1      │ byte │
  ├───────────────┼──────────────────┼──────┤
  │ DEFAULT_ALLOW │ POLICIES_APPLIED │ 4    │
  ├───────────────┼──────────────────┼──────┤
  │ DEFAULT_ALLOW │ DEFAULT_ALLOW    │ 2    │
  ├───────────────┼──────────────────┼──────┤
  │ DEFAULT_DENY  │ POLICIES_APPLIED │ 7    │
  ├───────────────┼──────────────────┼──────┤
  │ DEFAULT_DENY  │ DEFAULT_ALLOW    │ 7    │
  └───────────────┴──────────────────┴──────┘

Deleting the CNP moves entry 1 from POLICIES_APPLIED to DEFAULT_ALLOW. In standard mode that is 4 to 2, so the byte changes and the connection is re-evaluated and dropped. In strict mode it is 7 to 7, so the datapath returns BPF_OK without calling evaluateFlow, and the connection keeps flowing under a policy that no longer exists. There is a diff in standard mode vs strict mode evaluation here

The same collapse hides three other tightenings, all requiring strict mode plus a CNP: removing the last NetworkPolicy, adding a NetworkPolicy, and removing both at once. Four combinations are affected.

Test

Same operation both runs (delete the CNP), same long-lived TCP connection pinging every 3s, same node and agent. The only difference is entry 0: strict mode leaves it at DEFAULT_DENY, while a NetworkPolicy on the pod holds it at POLICIES_APPLIED.

  strict, entry0=DEFAULT_DENY (byte 7 -> 7)
    14:35:07  CNP deleted
    14:37:46  seq 69 OK        69 OK / 0 FAIL, still flowing after 2m39s
              new connections DENIED throughout

  standard + NetworkPolicy, entry0=POLICIES_APPLIED (byte 7 -> 5)
    14:49:59  seq 5 OK
    14:50:01  CNP deleted
    14:50:07  seq 6 FAIL       connection dropped in ~6s

Fix:

paramater ordering need to be fixed here

__u8 ct_pod_state_val = GET_CT_VAL(clusterpolicy_pst->state, pst->state);