accurately detect default gateways by aojea · Pull Request #153 · kubernetes-sigs/dranet

aojea · 2026-04-16T09:52:30Z

Address several critical edge cases in default interface detection and introduces a robust, rootless networking test framework.

Point-to-Point Interfaces: Removed the r.Gw == nil check. Previously, this caused the agent to completely ignore active VPNs and tunnels (like Wireguard or tun/tap) because P2P links route directly out of the device without a Gateway IP.
Route Metrics: The kernel relies on metrics (Priority) to determine the active route in a multi-WAN setup. The old code ignored this, returning all interfaces. It now correctly isolates the lowest metric for IPv4 and IPv6 independently, while preserving ECMP support.
Multipath Link Lookup: Fixed a bug where multipath routes were queried using the parent route's link index (which is often 0) rather than the nexthop's link index.

Address several critical edge cases in default interface detection and introduces a robust, rootless networking test framework. 1. Point-to-Point Interfaces: Removed the `r.Gw == nil` check. Previously, this caused the agent to completely ignore active VPNs and tunnels (like Wireguard or tun/tap) because P2P links route directly out of the device without a Gateway IP. 2. Route Metrics: The kernel relies on metrics (Priority) to determine the active route in a multi-WAN setup. The old code ignored this, returning all interfaces. It now correctly isolates the lowest metric for IPv4 and IPv6 independently, while preserving ECMP support. 3. Multipath Link Lookup: Fixed a bug where multipath routes were queried using the parent route's link index (which is often 0) rather than the nexthop's link index. Signed-off-by: Antonio Ojea <aojea@google.com>

aojea · 2026-04-16T09:54:25Z

/assign @gauravkghildiyal @dkennetzoracle @tamilmani1989

k8s-ci-robot · 2026-04-16T09:54:29Z

@aojea: GitHub didn't allow me to assign the following users: dkennetzoracle, tamilmani1989.

Note that only kubernetes-sigs members with read permissions, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time.
For more information please see the contributor guide

Details

In response to this:

/assign @gauravkghildiyal @dkennetzoracle @tamilmani1989

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

dkennetzoracle · 2026-04-16T16:25:43Z

-// as default gateways in the main routing table. It identifies these by querying
-// the main routing table for routes with an unspecified destination (0.0.0.0/0
-// for IPv4 or ::/0 for IPv6).
+// as active default gateways in the main routing table, respecting route metrics.


just a note / side-effect, any route matching Dst == 0.0.0.0/0 or ::/0 in the main table now classifies the link as an uplink, which is a new semantic and probably worth a comment?

is that heuristic not correct?

it is the same behavior as before, isn;t it?
it just expands the logic to take into consideration the weights, but previous logic about the 0.0.0.0 and ::/0 remains the same

It is the same logic, but the route no longer has to be a gateway. Any route with scope=link for example but with no gateway but still matches 0.0.0.0/0 or ::/0 will also match here.

+1 on semantic broadening — "default route with a gateway" to "any default route"

what is the difference, from the routing perspective the fact of having a next hop direction or just send to via the corresponding interface is not important, what is important is that any packet in the host will go through that specific interface ... that is why we don't want to expose it in the ResourceSlice

I don't feel super strongly about it, (to me) it just adds clarity.

dkennetzoracle · 2026-04-16T16:27:38Z

@@ -0,0 +1,102 @@
+package testutils


do you need a license header? I see them everywhere else.

yes, good catch

dkennetzoracle · 2026-04-16T18:05:36Z

+			},
+			expectedResult: sets.New[string]("eth0", "eth2"),
+		},
+		{


I'd add a test for Mixed-family default on a single interface. Basically, when you merge sets for ipv4 and ipv6 this asserts that the merge behaves (dual stacking an interface).

{ name: "Same interface wins both families", setupRoutes: func() error { if err := netlink.RouteAdd(&netlink.Route{Family: netlink.FAMILY_V4, Dst: defaultIPv4, Gw: gwIPv4, LinkIndex: links["eth0"].Attrs().Index, Priority: 100, Table: unix.RT_TABLE_MAIN}); err != nil { return err } return netlink.RouteAdd(&netlink.Route{Family: netlink.FAMILY_V6, Dst: defaultIPv6, Gw: gwIPv6, LinkIndex: links["eth0"].Attrs().Index, Priority: 100, Table: unix.RT_TABLE_MAIN}) }, expectedResult: sets.New[string]("eth0"), },

dkennetzoracle · 2026-04-16T18:23:35Z

+				return netlink.RouteAdd(&netlink.Route{Family: netlink.FAMILY_V4, Dst: defaultIPv4, Gw: gwIPv4, LinkIndex: links["eth0"].Attrs().Index, Priority: 100, Table: unix.RT_TABLE_MAIN})
+			},
+			expectedResult: sets.New[string]("wg0"),
+		},


Would it be worth also having a pinned Priority: 0 for P2P case?

You re using Prio 50 for wg0 and this exercises the GW == nil removal as well as the the lowest metric wins logic. However, it doesn't exercise a specific case where a P2P route has Priority: 0 which is the ipv4 default for a route installed without an explicit metric. This case would just lock in that 0 is a real comparable value.

OK with not doing it, it just feels a bit more explicit and defensive.

{ name: "P2P default with Priority 0 wins over gateway'd default", setupRoutes: func() error { if err := netlink.RouteAdd(&netlink.Route{Family: netlink.FAMILY_V4, Dst: defaultIPv4, LinkIndex: links["wg0"].Attrs().Index, Priority: 0, Scope: netlink.SCOPE_LINK, Table: unix.RT_TABLE_MAIN}); err != nil { return err } return netlink.RouteAdd(&netlink.Route{Family: netlink.FAMILY_V4, Dst: defaultIPv4, Gw: gwIPv4, LinkIndex: links["eth0"].Attrs().Index, Priority: 100, Table: unix.RT_TABLE_MAIN}) }, expectedResult: sets.New[string]("wg0"), },

dkennetzoracle · 2026-04-16T18:33:37Z

+// It caches the result after the first check.
+func IsSupported() bool {
+	checkOnce.Do(func() {
+		cmd := exec.Command("sleep", "1")


A small nit: this requires sleep to be on the system in PATH. If it's not, you'd get an error and it would be because of a missing binary not because user namespaces are not supported. Could check it?

maybe exec.Command(os.Args[0], "-test.run=^$") to avoid the external sleep dependency

dkennetzoracle · 2026-04-16T18:36:10Z

+	cmd.Args = []string{os.Args[0], "-test.run=" + t.Name() + "$", "-test.v=true"}
+
+	for _, arg := range os.Args {
+		if strings.HasPrefix(arg, "-test.testlogfile=") {


would there be any reason to add -test.timeout here? Any of these long?

dkennetzoracle

Some nits / questions, but I think it's a great add and a correctness fix. I'm not 100% sure it fixes the NonUplinkChecker situation on #138 on OKE, I'd need to double check. RA injects IPv6 defaults for us. If they share a metric (which I think they do) I still need something like NonUplinkChecker. If they don't I can drop NonUplinkChecker.

This definitely addresses one of the sub-issues in that PR, and is correct. LGTM!

gauravkghildiyal · 2026-04-16T18:45:26Z

I'm not 100% sure it fixes the NonUplinkChecker situation on #138 on OKE, I'd need to double check.

@dkennetzoracle I'm very much interested in hearing about this after you've had the opportunity to try this out

dkennetzoracle · 2026-04-16T18:53:03Z

@gauravkghildiyal - I will let you know! It's very hard for me to get access to GB200+ so I need to make the most of my time on them haha. I'm pretty sure each NIC gets a default route for ipv6 and these use the kernel's RA priority (so 1024) so we wouldn't be able to differentiate the 8 RDMA NICs from each other by metric. They all look like peer's from he kernel's pov.

So they'd all get excluded still here, haha. But it should work for the Azure case

anson627 · 2026-04-16T21:07:32Z

I just verified this PR on AWS p4d.24large, the primary NiC with default gateway/route (e.g. ens32) is excluded, and 4 EFA/ENA devices are properly included in the resource slice:

Device	Type	Interface	PCI Address	NUMA	IP
pci-0000-10-01-0	ENA	ens33	0000:10:01.0	0	192.168.157.168/19
pci-0000-10-1b-0	EFA	—	0000:10:1b.0	0	rdmap16s27
pci-0000-20-01-0	ENA	ens65	0000:20:01.0	0	192.168.155.210/19
pci-0000-20-1b-0	EFA	—	0000:20:1b.0	0	rdmap32s27
pci-0000-90-01-0	ENA	ens129	0000:90:01.0	1	192.168.145.129/19
pci-0000-90-1b-0	EFA	—	0000:90:1b.0	1	rdmap144s27
pci-0000-a0-01-0	ENA	ens161	0000:a0:01.0	1	192.168.132.181/19
pci-0000-a0-1b-0	EFA	—	0000:a0:1b.0	1	rdmap160s27

anson627 · 2026-04-16T22:05:38Z

verified this PR on Azure GB300, the primary NiC (eth0) with default gateway is excluded, while Azure accelerated networking VF (e.g. enP8051s1) together with other mlx5_* NiCs are included

#	Name	RDMA Device	PCI Address	NUMA	Vendor	RDMA
1	pci-0101-00-00-0	mlx5_0	0101:00:00.0	0	Mellanox	true
2	pci-0102-00-00-0	mlx5_1	0102:00:00.0	0	Mellanox	true
3	pci-0103-00-00-0	mlx5_2	0103:00:00.0	1	Mellanox	true
4	pci-0104-00-00-0	mlx5_3	0104:00:00.0	1	Mellanox	true
5	pci-1f73-00-02-0	—	1f73:00:02.0	0	Mellanox	true

/lgtm

k8s-ci-robot · 2026-04-16T22:06:24Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: anson627, aojea, dkennetzoracle

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [aojea]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

gauravkghildiyal · 2026-04-16T22:09:40Z

Holding to have time to go through this and allow resolution of a few comments.

/hold

aojea · 2026-04-17T11:33:20Z

RA injects IPv6 defaults for us. If they share a metric (which I think they do) I still need something like NonUplinkChecker. If they don't I can drop NonUplinkChecker.

From a kubernetes architectural and security perspective, we like to have certain guardrails so users can not shoot themselves on the feet. Any logic that allows an interface with a default route to be unmounted or modified based on an opportunistic RA injection is very risky.:

In the Kubernetes networking model, there is typically no difference between the control plane and the dataplane traffic. If an interface intended for the dataplane receives a default route via RA, it will be 'hijacking' control plane traffic, so the kubelet and pods trying to connect to the apiserver will fail (unless this secondary interface has access too, but this looks a very convoluted setup with routing loops). This will effectively disconnect the entire node from the cluster.
We already were hit by security issues "abusing" RA, so right now is RA processing disabled by default in all container interfaces, the same attack can be performed at the node level

A vulnerability was found in all versions of containernetworking/plugins before version 0.8.6, that allows malicious containers in Kubernetes clusters to perform man-in-the-middle (MitM) attacks. A malicious container can exploit this flaw by sending rogue IPv6 router advertisements to the host or other containers, to redirect traffic to the malicious container. (CVE-2020-10749)

I still think we need a better mechanism for improving filtering, so I encourage you to check @gauravkghildiyal proposal in #152 , that I think will be able to accomodate the NonUplinkChecker functionality perfectly

k8s-ci-robot added the kind/bug Categorizes issue or PR as related to a bug. label Apr 16, 2026

k8s-ci-robot requested review from MikeZappa87 and anson627 April 16, 2026 09:52

k8s-ci-robot added approved Indicates a PR has been approved by an approver from all required OWNERS files. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Apr 16, 2026

aojea mentioned this pull request Apr 16, 2026

Feat: Add real topology data and Working ipv6. #138

Open

k8s-ci-robot assigned gauravkghildiyal Apr 16, 2026

dkennetzoracle reviewed Apr 16, 2026

View reviewed changes

dkennetzoracle approved these changes Apr 16, 2026

View reviewed changes

k8s-ci-robot assigned anson627 Apr 16, 2026

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Apr 16, 2026

anson627 approved these changes Apr 16, 2026

View reviewed changes

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Apr 16, 2026

Conversation

aojea commented Apr 16, 2026

Uh oh!

aojea commented Apr 16, 2026

Uh oh!

k8s-ci-robot commented Apr 16, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dkennetzoracle left a comment

Choose a reason for hiding this comment

Uh oh!

gauravkghildiyal commented Apr 16, 2026

Uh oh!

dkennetzoracle commented Apr 16, 2026

Uh oh!

anson627 commented Apr 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

anson627 commented Apr 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

k8s-ci-robot commented Apr 16, 2026

Uh oh!

gauravkghildiyal commented Apr 16, 2026

Uh oh!

aojea commented Apr 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

anson627 commented Apr 16, 2026 •

edited

Loading

anson627 commented Apr 16, 2026 •

edited

Loading

aojea commented Apr 17, 2026 •

edited

Loading