Direct path periodically drops to relay every ~50-75s behind CGNAT with absolute UDP mapping timeout (T-Mobile) — make-before-break path rotation?

[huangyingw]

Setup

• Server: Raspberry Pi on residential fiber (public IPv4, symmetric NAT router, no UPnP), iroh 1.0.2 / noq 1.0.1, fixed UDP port, Builder::external_addr + pkarr publishing with an unfiltered-ish addr filter (so clients do get direct candidates).
• Client: iOS (iroh-ffi 1.1.0) on T-Mobile US cellular (CGNAT, 172.56.0.0/12 pool).
• Holepunching succeeds: connections reach SELECTED direct within ~1s of establishment. (This is the same deployment as my comment on Hole punching does not work when server side is behind symmetric NAT #4134 — direct now works after we started publishing candidates.)

Observed: direct path drops back to relay every ~50–75s, then re-punches

Server-side Connection::path_events() log for 4 consecutive sessions (timestamps relative):

SELECTED direct  →  +50s closed → SELECTED relay → +1s   SELECTED direct (re-punched)
SELECTED direct  →  +48s closed → SELECTED relay → +26s  SELECTED direct
SELECTED direct  →  +57s closed → SELECTED relay → +4s   SELECTED direct
SELECTED direct  →  +75s closed → SELECTED relay → (conn ended)

The pattern repeats for the lifetime of every connection: during continuous media streaming this means a relay (rate-limited) window roughly once per minute.

Analysis

• Per-path 5s heartbeats (HEARTBEAT_INTERVAL) are present, so this is not an idle-traffic problem.
• The timing strongly matches CGNAT absolute UDP mapping timeout / port rebinding (T-Mobile is commonly reported at ~30–60s, regardless of traffic). Once the mapping is re-bound, the old 4-tuple is a black hole; iroh correctly detects it after PATH_MAX_IDLE_TIMEOUT (15s), falls back to relay, and re-punches (5s attempt interval) — i.e. detection+rebuild is working as designed, worst-case ~26s on relay.
• Key point: with an absolute timeout, keep-alives cannot save the mapping, and while the mapping is still alive the same internal source port keeps mapping to the same external tuple — so "refreshing" over the same socket can't produce a fresh mapping either. Surviving this NAT class seems to require make-before-break with a tuple change: proactively establish a new direct path (new local source port via rebind, or a different remote port candidate) shortly before the old mapping's expected death, switch over, and let the old path drain.

Questions / proposals

1. Is there an intended mechanism for this scenario in the current multipath design (noq)? If not, would you be open to:
   a. Proactive path rotation: when a direct path's age approaches an (adaptive or configurable) threshold, start a holepunch round for a fresh tuple while the old path is still selected — zero relay window on switch; or at least
   b. Faster rebuild knobs: PATH_MAX_IDLE_TIMEOUT / HOLEPUNCH_ATTEMPTS_INTERVAL (and friends) are pub(crate) consts today; making them configurable would let deployments on high-churn NATs shrink the relay window (15s+ → a few s) without protocol changes.
2. We're happy to contribute either (and can validate against this real T-Mobile CGNAT deployment with server-side path-event data). Which direction fits your roadmap?

(Server-side per-path close stats — rtt/tx/rx/lost/blackhole counters — are being collected on our side and can be attached if useful.)

[matheus23]

direct now works after we started publishing candidates

Does this mean you have made some code changes? Or what exactly have you changed since your original comment in the other issue?

[flub]

Thanks for this report, this is very valuable information. We have a known issue that we need to detect paths that became unusable faster which will go a long way towards helping this.

Proactive opening new direct paths is pretty hard right now. Because you still have a working mapping the path would just stay there. I'd like to look a bit deeper into what your ISP does to udp mappings here.

[huangyingw]

@matheus23 No iroh code changes — public API only, two things on our side:

1. Server (embedding): our node publishes its reachable addr via Builder::external_addr(...), but we found those never appeared in the pkarr/DNS record — PkarrPublisher's default AddrFilter::relay_only() filters them out. We now build the publisher with a custom filter (relay + global-unicast IPs only, so no private/LL/ULA leakage). That was the "started publishing candidates" change: dial-by-key clients then actually receive direct candidates at resolve time. (Docs suggestion: a note on external_addr that the default publisher filter will drop it from discovery would save integrators some debugging — or perhaps explicitly-configured external addrs could bypass the default filter, since setting them is already an explicit opt-in.)
2. Client: iroh-ffi 1.0.0 → 1.1.0 (pulling iroh 1.0.2 / noq 1.0.1).

@flub Per-path close stats from the server side (same T-Mobile deployment, cellular direct paths):

closed addr=ip:172.56.234.252:21449 stats[rtt=59ms  tx=172540396B/118984pkt rx=1886100B/43003pkt lost=287pkt  blackholes=0]
closed addr=ip:172.56.234.168:38340 stats[rtt=734ms tx=117291718B/80870pkt  rx=1090343B/24704pkt lost=195pkt  blackholes=0]
closed addr=relay:usw1-1            stats[rtt=306ms tx=55903557B/39518pkt   rx=165900B/2746pkt   lost=1381pkt blackholes=0]

Two observations that may be useful:

• black_holes_detected stays 0 even for the direct paths that die in the ~50–75s pattern — so the current blackhole detector doesn't classify this death mode, and detection falls back to the idle timeout. That seems consistent with your known issue about detecting unusable paths faster.
• The CGNAT pool address itself changes across sessions (172.56.234.252 → 172.56.234.168 → earlier 172.56.235.186), supporting the rebinding interpretation.

On the proactive side: understood that with a still-working mapping, a "new" path over the same socket is just the same tuple. Would a periodic/pre-expiry local rebind (new source socket → new mapping, punch, switch, drop old) be a direction you'd consider at the transports layer, or is that off the table? Happy to run controlled probes against this ISP (e.g. UDP echo with varying inter-packet gaps to pin down the exact mapping lifetime/absoluteness) if that data helps.

[huangyingw]

Got a valid measurement now (client actually routed through the T-Mobile cellular CGNAT this time). 90s of continuous STUN Binding every 2s to a fixed STUN server (so the mapping is never idle), observing the external XOR-MAPPED-ADDRESS. Redacted the first two octets of the CGNAT address.

[ 0.0s] ext=x.x.234.157:48688   (initial mapping)
[ 6.1s] ext=x.x.234.157:48688
[ 8.0s] (1 packet lost)
[10.1s] ext=x.x.133.14:23141   ← REBIND: both IP AND port changed
[12s..88s] ext=x.x.133.14:23141   (stable for the remaining ~80s)

Two takeaways that I think refine the picture:

1. The external mapping was re-bound mid-session under continuous traffic (~8s in), and crucially the external IP changed too (234.157 → 133.14), not just the port. After that single rebind it was then stable for 80s. So it's not a clean clockwork "~50s absolute timeout" — it looks more like occasional, unpredictable re-mapping of the external endpoint even while the flow is active. That matches the variable drop intervals I reported in the original issue (48/50/57/75s rather than a fixed period).

2. Because the rebind changes the whole external tuple (IP+port), a peer that had holepunched to the old tuple is now black-holed, and there's nothing predictable to aim a new path at — so any make-before-break here would need to run a full re-punch through the coordination channel to discover the new tuple, not a port-prediction/rebind-to-known-port shortcut.

So combining with your "detect unusable paths faster" known issue: faster detection shrinks the relay window after a rebind, but fully avoiding it would need proactively establishing a fresh path before the CGNAT re-maps — which, given the re-map is aperiodic and changes the IP, seems genuinely hard as you noted.

Happy to gather more (e.g. longer runs to characterize the rebind interval distribution, or idle-timeout sweeps) if a particular cut of data would help. The little STUN probe that produced this is ~200 lines of stdlib Python if it's useful to reproduce on other carriers.