set: fix byte order metadata for host-endian types and map data

[anthonyrisinger]

Fix two issues with set/map userdata byte order metadata that cause nft list ruleset to misinterpret element keys and data values on little-endian systems. The kernel-level data is always correct — these are display/round-trip issues affecting the userdata TLVs that the nft CLI uses to reconstruct human-readable output.

Bug 1: Anonymous sets ignore explicit KeyByteOrder

Anonymous/constant/interval sets unconditionally emit KEYBYTEORDER=2 (big-endian), ignoring the KeyByteOrder field. The nft C tool always uses the key expression's actual byte order (mnl.c:mnl_nft_set_add), not a blanket default.

For host-endian types like mark_type (BYTEORDER_HOST_ENDIAN) and ifname_type (BYTEORDER_HOST_ENDIAN), this causes nft list ruleset to misinterpret the element data on LE systems:

• Mark sets: meta mark { 0x00000000, 0x01000000 } instead of { 0x00000000, 0x00000001 }
• Interface name vmaps: iifname vmap { "" : jump chain1 } instead of { "eth0" : jump chain1 }

The existing comment "Semantically useless - kept for binary compatability with nft" is incorrect — KEYBYTEORDER IS semantically meaningful for host-endian types. The nft C tool uses it in netlink_delinearize_setelem to decide whether to call mpz_switch_byteorder on element keys.

Fix: When the anonymous/constant/interval condition fires, check if KeyByteOrder is explicitly set to NativeEndian and emit 1 (host) instead of 2 (big). Falls back to the original big-endian default when KeyByteOrder is unset (zero value), preserving backwards compatibility.

Bug 2: Maps never emit DATABYTEORDER

The NFTNL_UDATA_SET_DATABYTEORDER constant exists in userdata.go but is never used. The Set struct has no DataByteOrder field, and the userdata construction has no code path that emits this TLV.

The nft C tool emits DATABYTEORDER for all datamaps:

if (set_is_datamap(set->flags))
    nftnl_udata_put_u32(udbuf, NFTNL_UDATA_SET_DATABYTEORDER,
                         set->data->byteorder);

Without it, nft list ruleset uses BYTEORDER_INVALID for map data, skips the mpz_switch_byteorder call, and displays native-endian values as byte-swapped on LE systems:

# Expected:
elements = { ... : 0x00000001 }

# Without DATABYTEORDER on LE:
elements = { ... : 0x01000000 }

Fix: Add DataByteOrder field to Set struct and emit NFTNL_UDATA_SET_DATABYTEORDER in the userdata construction for maps when set.

Discovery context

Found while debugging a WireGuard mesh overlay that uses meta mark sets and maps for zone-based connection latching. The incorrect display made it extremely difficult to diagnose whether mark values were being set correctly, since nft list ruleset showed byte-swapped values but the kernel data was actually correct (verified via nft --debug=netlink list ruleset).

Testing

Verified on linux/arm64 (little-endian) that after this patch:

• Anonymous TypeMark sets display correct values (0x00000001 not 0x01000000)
• Map data with TypeMark displays correct values
• nft --debug=netlink list ruleset shows identical kernel-level bytes before and after (no functional change)
• GOOS=linux go build . and GOOS=linux go vet . pass cleanly

[anthonyrisinger]

Company decided they wanted the PR from here—apologies for the noise from the last two closed!—I couldn't figure out how to open #353 again; it was the same exact commit on all three.

[nickgarlis]

I took a look through this and left some suggestions / personal preferences. @stapelberg I don't mean to step on your toes. Just thought those comments might be helpful.

[anthonyrisinger]

Let me know if there's any other changes—I simplify tests with a round-trip through both encode/decode paths.

[anthonyrisinger]

Should be all set! Validated with a Docker container:

$ docker run --rm --privileged -v ~/devel/nftables:/work -w /work golang:1.24-bookworm bash -lc '/usr/local/go/bin/go test -count=5 -run "^TestSetByteOrderRoundTrip$" -run_system_tests .'

go: downloading github.com/mdlayher/netlink v1.8.1-0.20251028132421-dcc6cab9a6eb
go: downloading golang.org/x/sys v0.35.0
go: downloading github.com/vishvananda/netns v0.0.4
go: downloading golang.org/x/net v0.43.0
go: downloading github.com/mdlayher/socket v0.5.1
go: downloading golang.org/x/sync v0.6.0
PASS
ok  	github.com/google/nftables	0.022s

[nickgarlis]

LGTM