~600 Mbps throughput when AQC113 is behind Thunderbolt 4 dock + Thunderbolt 3 hop; full 10 GbE direct on Windows

[rajil]

Summary

On Linux 6.18.40, an AQC113-based Thunderbolt 3 NIC adapter
(Ubiquiti UACC-Adapter-RJ45-USBC-10GE) achieves ~600 Mbps when
connected through a Dell WD22TB4 Thunderbolt 4 dock. The same adapter
achieves full 10 GbE when plugged directly into the laptop's Thunderbolt
port. Windows achieves full 10 GbE through the dock. PCIe topology and
link speeds are identical between Linux and Windows in both scenarios,
pointing to a driver-level issue.

Hardware

• Laptop: Framework 13 AMD (Strix Halo / Ryzen AI)
• USB4: AMD USB4 Router [1022:151c / 1022:151d]
• Host TB: Intel Goshen Ridge TB4 [8086:0b26 rev 03]
• Dock: Dell WD22TB4 Thunderbolt 4 Dock
• Adapter: Ubiquiti UACC-Adapter-RJ45-USBC-10GE
  (Intel JHL7440 TB3 bridge [8086:15ef rev 06] +
  Aquantia AQC113 [1d6a:04c0 rev 03])
• Switch: Connected to 10 GbE switch

Kernel and driver

Kernel: 6.18.40
Driver: in-tree atlantic (intree: Y)
Module params: aq_itr, aq_itr_tx, aq_itr_rx (no other params exposed)

dmesg

[ 5.427263] thunderbolt 1-2: Dell WD22TB4 Thunderbolt Dock
[ 6.158888] thunderbolt 1-0:2.1: new retimer found, vendor=0x1da0 device=0x8833
[ 6.994932] thunderbolt 1-2:1.1: new retimer found, vendor=0x1da0 device=0x8833
[ 7.521693] thunderbolt 1-502: Ubiquiti Inc. UACC-Adapter-RJ45-USBC-10GE
[ 244.475274] atlantic 0000:84:00.0: enabling device (0000 -> 0002)
[ 244.559142] atlantic: Detect ATL2FW 1030025
[ 244.738977] atlantic 0000:84:00.0 enp132s0: renamed from eth1
[ 253.006960] atlantic 0000:84:00.0 enp132s0: atlantic: link change old 0 new 10000

The NIC reports a 10 Gbps Ethernet link in both direct and dock
scenarios. The limitation is not at the network layer.

PCIe topology

PCIe path to NIC - direct connection (Linux and Windows identical):

AMD USB4 Bridge 00:01.1 x1 @ 2.5 GT/s
JHL7440 root 01:00.0 x4 @ 2.5 GT/s (TB tunnel)
JHL7440 port 1 02:01.0 x4 @ 8.0 GT/s
AQC113 NIC 03:00.0 x4 @ 8.0 GT/s (downgraded from 16)

PCIe path to NIC - via dock (Linux and Windows identical):

AMD USB4 Bridge 00:01.2 x1 @ 2.5 GT/s
Goshen Ridge root 60:00.0 x4 @ 2.5 GT/s
Goshen Ridge port 2 61:02.0 x4 @ 2.5 GT/s
JHL7440 root 82:00.0 x4 @ 2.5 GT/s (TB tunnel)
JHL7440 port 1 83:01.0 x4 @ 8.0 GT/s
AQC113 NIC 84:00.0 x4 @ 8.0 GT/s (downgraded from 16)

The NIC's own PCIe link (x4 @ 8 GT/s between JHL7440 port 1 and the
AQC113) is identical across all four combinations of OS and connection
method. The extra hop in the dock path is the Goshen Ridge TB4 switch,
but this is also present and identical in the working Windows dock case.

lspci -vvnn for AQC113 (dock scenario)

84:00.0 Ethernet controller [0200]: Aquantia Corp. AQtion AQC113
NBase-T/IEEE 802.3an [1d6a:04c0] (rev 03)
LnkCap: Port #0, Speed 16GT/s, Width x4, ASPM not supported
LnkSta: Speed 8GT/s (downgraded), Width x4
LnkCtl2: Target Link Speed: 16GT/s
MaxPayload 128 bytes, MaxReadReq 512 bytes
Kernel driver in use: atlantic
Kernel modules: atlantic

Comparison across all four scenarios

Scenario OS JHL7440 port 1 AQC113 LnkSta Throughput
Direct Linux x4 @ 8 GT/s x4 @ 8 GT/s 10 GbE
Direct Windows x4 @ 8 GT/s x4 @ 8 GT/s 10 GbE
Via dock Windows x4 @ 8 GT/s x4 @ 8 GT/s 10 GbE
Via dock Linux x4 @ 8 GT/s x4 @ 8 GT/s ~600 Mbps

The hardware PCIe configuration is identical in all four cases. The
only variable that correlates with the degraded throughput is Linux
via the dock.

Things already tried

• ethtool -C enp132s0 rx-usecs 0 tx-usecs 0 (disable coalescing)
• ethtool -G enp132s0 rx 4096 tx 4096 (larger ring buffers)
• modprobe atlantic aq_itr=0 (disable ITR via param)

None of the above recovered throughput.

Questions

1. Is there a known issue with the atlantic driver when the AQC113 is
   behind multiple PCIe switches, specifically a TB4 + TB3 bridge chain?

2. Could MaxPayload size negotiation across six bridge hops cause a
   throughput ceiling? The NIC supports 512 bytes but is operating at
   128 bytes. Would it help to force a higher value via setpci?

3. Is there additional debug output (ring stats, DMA stats, firmware
   counters) that would help isolate whether the bottleneck is in the
   TX path, RX path, or interrupt handling?

Attachments

dock-output.txt - full lspci -tv and lspci -vvnn, dock scenario
direct-output.txt - full lspci -tv and lspci -vvnn, direct scenario
windows-dock.txt - HWiNFO64 PCI report, dock scenario (Windows)
windows-direct.txt - HWiNFO64 PCI report, direct scenario (Windows)

windows-direct.txt
windows-dock.txt

direct-output.txt
dock-output.txt

[rajil]

Summary

When an AQC113-based Thunderbolt 3 NIC adapter is connected through a
Thunderbolt 4 dock, the atlantic driver's TX queues stall repeatedly due
to TX ring exhaustion. This is caused by PCIe completion latency being
too high through the multi-hop Thunderbolt bridge chain. The result is
~1 Gbps throughput instead of the expected 10 GbE. Increasing the TX
ring size to the hardware maximum partially mitigates the issue but does
not resolve it. Direct connection (single Thunderbolt hop) produces zero
queue stalls and full 10 GbE. Windows achieves full 10 GbE through the
same dock with the same hardware.

Hardware

• Laptop: Framework 13 AMD (Strix Halo / Ryzen AI)
• Dock: Dell WD22TB4 Thunderbolt 4 Dock
• Adapter: Ubiquiti UACC-Adapter-RJ45-USBC-10GE
  (Intel JHL7440 TB3 bridge [8086:15ef rev 06] +
  Aquantia AQC113 [1d6a:04c0 rev 03])
• Kernel: 6.18.40
• Driver: in-tree atlantic (intree: Y)

PCIe path

Direct connection (1 Thunderbolt hop):

AMD USB4 → JHL7440 TB3 root → JHL7440 internal port → AQC113

Via dock (2 Thunderbolt hops + Goshen Ridge switch):

AMD USB4 → Goshen Ridge TB4 → Dell WD22TB4 → JHL7440 TB3 root
→ JHL7440 internal port → AQC113

The AQC113 negotiates x4 at 8 GT/s to the JHL7440 internal port in
both cases. The PCIe link speed and width at the NIC are identical.
The additional latency comes from the extra bridge hops in the dock path.

Reproduction

1. Connect Ubiquiti UACC-Adapter-RJ45-USBC-10GE to a Dell WD22TB4 dock
2. Connect dock to laptop via Thunderbolt 4
3. Run iperf3 throughput test (iperf3 -c -t 30 -P 4)
4. Capture ethtool stats mid-test:
   ethtool -S | grep -iv ' 0$'

Evidence

Direct connection (eno1, bus 03:00.0, TX ring = 4096 default)

OutOctets in 10 seconds: 14,566,654,464 (~11.7 Gbps)
Queue Restarts: 0 on all queues

Queue[0] OutPackets: 5,018,773 Polls: 48,936 Restarts: 0
Queue[7] OutPackets: 4,981,037 Polls: 51,047 Restarts: 0

Via dock (enp101s0, bus 84:00.0, TX ring = 4096)

OutOctets in 10 seconds: 1,247,335,030 (~1.0 Gbps)
Queue Restarts: ~2000 per active TX queue

Queue[2] OutPackets: 270,110 Polls: 2,242 Restarts: 1,981
Queue[4] OutPackets: 275,090 Polls: 2,296 Restarts: 2,153
Queue[6] OutPackets: 279,326 Polls: 2,502 Restarts: 2,193

Via dock (enp101s0, TX ring increased to 8184 maximum)

OutOctets in 10 seconds: ~2,740,000,000 (~2.2 Gbps)
Queue Restarts: ~2500 per active TX queue

Queue[0] OutPackets: 344,234 Polls: 2,764 Restarts: 2,532
Queue[6] OutPackets: 339,328 Polls: 2,719 Restarts: 2,631

Summary table

Scenario TX ring Throughput Restarts/queue (10s)
Direct 4096 ~11.7 Gbps 0
Dock 4096 ~1.0 Gbps ~2000
Dock 8184 ~2.2 Gbps ~2500

Analysis

TX queue restarts (netdev_tx_queue_stopped / netdev_wake_queue) indicate
the TX descriptor ring is being exhausted faster than completions are
returned. In the dock scenario, PCIe completions must travel back through
the JHL7440 TB3 bridge, the Dell WD22TB4 dock, the Goshen Ridge TB4
switch, and the AMD USB4 bridge before reaching the CPU. This round-trip
latency is high enough that the TX ring fills up over 2000 times in 10
seconds on each active queue.

Increasing the TX ring to the hardware maximum (8184) improves throughput
from ~1 Gbps to ~2.2 Gbps but does not eliminate restarts. The per-queue
poll count in the dock scenario (2,242-2,764 polls per queue) is
dramatically lower than in the direct scenario (48,936-51,047 polls per
queue), confirming that TX completion interrupts are being delivered far
less frequently through the dock path.

Windows achieves full 10 GbE through the same dock, suggesting the issue
is specific to how the atlantic driver handles TX completion latency in
deep PCIe topologies rather than a hardware limitation.

Workaround (partial)

ethtool -G tx 8184

Increases throughput from ~1 Gbps to ~2.2 Gbps but does not fully
resolve the issue.

Suggested fix direction

The driver needs to handle high PCIe completion latency more gracefully
when the device is behind multiple PCIe switches. Possible approaches:

• More aggressive NAPI-based TX completion polling in high-latency
  conditions rather than relying on completion interrupts
• A module parameter or ethtool private flag to tune TX completion
  behaviour for deep PCIe topologies
• Adaptive detection of completion latency with automatic ring size
  or polling rate adjustment

Additional notes

• ethtool -c shows interrupt coalescing is active; disabling it via
  ethtool -C rx-usecs 0 tx-usecs 0 did not resolve the issue
• The AQC113 reports a 10 Gbps Ethernet link in both direct and dock
  scenarios; the limitation is entirely in the TX completion path
• Tested with kernel 6.18.40 in-tree atlantic driver

[rajil]

Update: root cause identified via ethtool -S

After filing this report I ran ethtool -S during an active iperf3 test
(-c -t 30 -P 4) in both scenarios. The dock scenario shows
thousands of TX queue restarts per 10 seconds. The direct connection
shows none.

Direct connection (eno1, TX ring = 4096):

OutOctets in 10s: 14,566,654,464 (~11.7 Gbps)
Queue Restarts: 0 on all queues
Queue[0] OutPackets: 5,018,773 Polls: 48,936 Irqs: ~48,936
Queue[7] OutPackets: 4,981,037 Polls: 51,047 Irqs: ~51,047

Via dock (enp101s0, TX ring = 4096):

OutOctets in 10s: 1,247,335,030 (~1.0 Gbps)
Queue[2] OutPackets: 270,110 Polls: 2,242 Restarts: 1,981
Queue[4] OutPackets: 275,090 Polls: 2,296 Restarts: 2,153
Queue[6] OutPackets: 279,326 Polls: 2,502 Restarts: 2,193

Via dock (enp101s0, TX ring increased to maximum 8184):

OutOctets in 10s: ~2,740,000,000 (~2.2 Gbps)
Queue[0] OutPackets: 344,234 Polls: 2,764 Restarts: 2,532
Queue[6] OutPackets: 339,328 Polls: 2,719 Restarts: 2,631

The TX ring exhaustion is directly visible in the out-of-tree driver's
additional Head/Tail stats:

Queue[4] TX Head: 93 TX Tail: 4048 (3955/4096 = 96% full)
Queue[1] TX Head: 1299 TX Tail: 1030 (wrapped, ~3827 pending)

The IRQ vs Poll ratio tells the story clearly:

Queues with restarts: ~40 IRQs, ~2800 Polls (70x more polls than IRQs)
Queues without restarts: IRQs and Polls are 1:1

TX completions are not being delivered via interrupt in time through the
multi-hop Thunderbolt bridge chain. NAPI polls repeatedly without finding
completions, the TX ring fills up, and the queue stalls thousands of
times per 10 seconds.

Increasing the TX ring to 8184 (ethtool -G tx 8184) improves throughput
from ~1 Gbps to ~2.2 Gbps but does not eliminate restarts. The
completion interrupt starvation persists regardless of ring size.

This was reproduced on both the in-tree driver (kernel 6.18.40) and the
latest out-of-tree AQtion driver.

The direct connection (single Thunderbolt hop) delivers completions fast
enough that the TX ring never exhausts. The dock path (AMD USB4 ->
Goshen Ridge TB4 -> Dell WD22TB4 -> JHL7440 -> AQC113) adds enough
PCIe round-trip latency that completion interrupts cannot keep up with
the TX submission rate.

Windows achieves full 10 GbE through the same dock, suggesting it
either handles TX completions differently or does not suffer the same
interrupt delivery latency in this topology.