Skip to content

Validate pinned BPF map schemas at startup #651

Description

@Pavani-Panakanti

Summary

At startup the agent decides whether to recreate its pinned BPF maps by comparing packaged binaries against the copies installed on the host. Nothing validates the schema of a map that is already pinned. If the agent restarts after binaries are copied but before recovery completed, it can adopt a pinned map whose value_size does not match what the userspace structs and the newly loaded programs expect, and it will keep doing so on every subsequent restart.

Current behaviour

checkAndUpdateBPFBinaries (pkg/ebpf/bpf_client.go:343) computes the three update flags purely from a byte comparison of the probe binaries:

isEqual := cmp.Equal(currentProbe, existingProbe)

Startup then runs in this order:

Line Step
bpf_client.go:168 compute ingress/egress/eventsUpdateRequired from binary bytes
bpf_client.go:178 cp.InstallBPFBinaries copies the new binaries to the host
bpf_client.go:188 recoverBPFState adopts whatever is pinned
bpf_client.go:203 if eventsUpdateRequired || (!isConntrackMapPresent || !isPolicyEventsMapPresent) recreates the global maps

Inside recoverBPFState, a pinned map is adopted on the strength of its pin path alone (bpf_client.go:419-421):

if globalMapName == CONNTRACK_MAP_PIN_PATH {
    log().Info("Conntrack Map is already present on the node")
    isConntrackMapPresent = true
    globalMaps.Store(globalMapName, globalMap)
}

There is no check of type, key size, or value size.

The failure

Binaries are copied at line 178, but the maps are not recreated until line 203. A crash, OOM kill, or node reboot in that window leaves the host with new binaries and an old map. On the next start:

  1. packaged and host binaries now match, so all three update flags are false
  2. the old map is still pinned, so isConntrackMapPresent is true
  3. the recreate branch at line 203 is skipped
  4. the agent adopts a map whose layout disagrees with the loaded programs

This does not self-heal. The binary comparison keeps returning equal on every restart, so the agent re-enters the same state indefinitely. Today the only way out is deleting the pin by hand or replacing the node.

This applied to all maps

The same window exists for every pinned map, and the recovery gates are equally schema-blind:

  • bpf_client.go:409if !updateEventsProbe adopts the global maps
    (aws_conntrack_map, policy_events)
  • bpf_client.go:434if !updateIngressProbe || !updateEgressProbe adopts the
    per-pod-identifier maps

That covers 8 pinned maps: the 2 global ones plus the 6 in
utils.NamespacedBPFMaps (ingress_map, egress_map, cp_ingress_map,
cp_egress_map, ingress_pod_state_map, egress_pod_state_map).

Proposed change

Extend the update trigger from "binary changed" to "binary changed or schema mismatch", keeping the existing flags as the mechanism so the remediation path is the normal upgrade path:

current:  updateRequired = binaryChanged
proposed: updateRequired = binaryChanged || schemaMismatch

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't workingenhancementNew feature or request

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions