Summary
At startup the agent decides whether to recreate its pinned BPF maps by comparing packaged binaries against the copies installed on the host. Nothing validates the schema of a map that is already pinned. If the agent restarts after binaries are copied but before recovery completed, it can adopt a pinned map whose value_size does not match what the userspace structs and the newly loaded programs expect, and it will keep doing so on every subsequent restart.
Current behaviour
checkAndUpdateBPFBinaries (pkg/ebpf/bpf_client.go:343) computes the three update flags purely from a byte comparison of the probe binaries:
isEqual := cmp.Equal(currentProbe, existingProbe)
Startup then runs in this order:
| Line |
Step |
bpf_client.go:168 |
compute ingress/egress/eventsUpdateRequired from binary bytes |
bpf_client.go:178 |
cp.InstallBPFBinaries copies the new binaries to the host |
bpf_client.go:188 |
recoverBPFState adopts whatever is pinned |
bpf_client.go:203 |
if eventsUpdateRequired || (!isConntrackMapPresent || !isPolicyEventsMapPresent) recreates the global maps |
Inside recoverBPFState, a pinned map is adopted on the strength of its pin path alone (bpf_client.go:419-421):
if globalMapName == CONNTRACK_MAP_PIN_PATH {
log().Info("Conntrack Map is already present on the node")
isConntrackMapPresent = true
globalMaps.Store(globalMapName, globalMap)
}
There is no check of type, key size, or value size.
The failure
Binaries are copied at line 178, but the maps are not recreated until line 203. A crash, OOM kill, or node reboot in that window leaves the host with new binaries and an old map. On the next start:
- packaged and host binaries now match, so all three update flags are
false
- the old map is still pinned, so
isConntrackMapPresent is true
- the recreate branch at line 203 is skipped
- the agent adopts a map whose layout disagrees with the loaded programs
This does not self-heal. The binary comparison keeps returning equal on every restart, so the agent re-enters the same state indefinitely. Today the only way out is deleting the pin by hand or replacing the node.
This applied to all maps
The same window exists for every pinned map, and the recovery gates are equally schema-blind:
bpf_client.go:409 — if !updateEventsProbe adopts the global maps
(aws_conntrack_map, policy_events)
bpf_client.go:434 — if !updateIngressProbe || !updateEgressProbe adopts the
per-pod-identifier maps
That covers 8 pinned maps: the 2 global ones plus the 6 in
utils.NamespacedBPFMaps (ingress_map, egress_map, cp_ingress_map,
cp_egress_map, ingress_pod_state_map, egress_pod_state_map).
Proposed change
Extend the update trigger from "binary changed" to "binary changed or schema mismatch", keeping the existing flags as the mechanism so the remediation path is the normal upgrade path:
current: updateRequired = binaryChanged
proposed: updateRequired = binaryChanged || schemaMismatch
Summary
At startup the agent decides whether to recreate its pinned BPF maps by comparing packaged binaries against the copies installed on the host. Nothing validates the schema of a map that is already pinned. If the agent restarts after binaries are copied but before recovery completed, it can adopt a pinned map whose
value_sizedoes not match what the userspace structs and the newly loaded programs expect, and it will keep doing so on every subsequent restart.Current behaviour
checkAndUpdateBPFBinaries(pkg/ebpf/bpf_client.go:343) computes the three update flags purely from a byte comparison of the probe binaries:Startup then runs in this order:
bpf_client.go:168ingress/egress/eventsUpdateRequiredfrom binary bytesbpf_client.go:178cp.InstallBPFBinariescopies the new binaries to the hostbpf_client.go:188recoverBPFStateadopts whatever is pinnedbpf_client.go:203if eventsUpdateRequired || (!isConntrackMapPresent || !isPolicyEventsMapPresent)recreates the global mapsInside
recoverBPFState, a pinned map is adopted on the strength of its pin path alone (bpf_client.go:419-421):There is no check of type, key size, or value size.
The failure
Binaries are copied at line 178, but the maps are not recreated until line 203. A crash, OOM kill, or node reboot in that window leaves the host with new binaries and an old map. On the next start:
falseisConntrackMapPresentistrueThis does not self-heal. The binary comparison keeps returning equal on every restart, so the agent re-enters the same state indefinitely. Today the only way out is deleting the pin by hand or replacing the node.
This applied to all maps
The same window exists for every pinned map, and the recovery gates are equally schema-blind:
bpf_client.go:409—if !updateEventsProbeadopts the global maps(
aws_conntrack_map,policy_events)bpf_client.go:434—if !updateIngressProbe || !updateEgressProbeadopts theper-pod-identifier maps
That covers 8 pinned maps: the 2 global ones plus the 6 in
utils.NamespacedBPFMaps(ingress_map,egress_map,cp_ingress_map,cp_egress_map,ingress_pod_state_map,egress_pod_state_map).Proposed change
Extend the update trigger from "binary changed" to "binary changed or schema mismatch", keeping the existing flags as the mechanism so the remediation path is the normal upgrade path: