Dell Pro Max with GB10 (NVIDIA Spark OEM model)

[geerlingguy]

Basic information

• Board URL (official): https://www.dell.com/en-us/shop/desktop-computers/dell-pro-max-with-gb10/spd/dell-pro-max-fcm1253-micro/xcto_fcm1253_usx
• Board purchased from: Provided by Dell, for testing
• Board purchase date: November 19, 2025
• Board specs (as tested): NVIDIA GB10 / 128 GB LPDDR5x / 4TB NVMe SSD
• Board price (as tested): $3998.99

Linux/system information

# output of `screenfetch`
                          ./+o+-       jgeerling@dell-gb10-1
                  yyyyy- -yyyyyy+      OS: Ubuntu 24.04 noble
               ://+//////-yyyyyyo      Kernel: aarch64 Linux 6.14.0-1013-nvidia
           .++ .:/++++++/-.+sss/`      Uptime: 34m
         .:++o:  /++++++++/:--:/-      Packages: 2339
        o:+o+:++.`..```.-/oo+++++/     Shell: dash
       .:+o:+o/.          `+sssoo+/    Disk: 36G / 3.6T (2%)
  .++/+:+oo+o:`             /sssooo.   CPU: ARM Cortex-X925 Cortex-A725 @ 20x 2.808GHz
 /+++//+:`oo+o               /::--:.   GPU: NVIDIA GB10
 \+/+o+++`o++o               ++////.   RAM: 3806MiB / 122506MiB
  .++.o+++oo+:`             /dddhhh.  
       .+.o+oo:.          `oddhhhh+   
        \+.++o+o``-````.:ohdhhhhh+    
         `:o+++ `ohhhhhhhhyo++os:     
           .o:`.syhhhhhhh/.oo++o`     
               /osyyyyyyo++ooo+++/    
                   ````` +oo+++o\:    
                          `oo++.      

# output of `uname -a`
Linux dell-gb10-1 6.14.0-1013-nvidia #13-Ubuntu SMP PREEMPT_DYNAMIC Wed Oct 29 06:01:19 UTC 2025 aarch64 aarch64 aarch64 GNU/Linux

Benchmark results

CPU

• Geekbench 6: (3123 single / 19708 multi - https://browser.geekbench.com/v6/cpu/15161960)
• 675.39 Gflops at 155W for 4.36 Gflops/W (geerlingguy/top500-benchmark HPL result)

Power

• Idle power draw (at wall): 31.5 W (1.3 W during shutdown, 38W with 200 Gbps link connected)
• Maximum simulated power draw (stress-ng --matrix 0): 127.1 W
• During Geekbench multicore benchmark: 114.8 W
• During top500 HPL benchmark: 155 W

Disk

ESL04TBTLCZ-27J4-TYN (Phison 4TB NVMe SSD)

Benchmark	Result
iozone 4K random read	80.51 MB/s
iozone 4K random write	409.89 MB/s
iozone 1M random read	3904.79 MB/s
iozone 1M random write	4938.67 MB/s
iozone 1M sequential read	3900.27 MB/s
iozone 1M sequential write	4955.19 MB/s

Network

iperf3 results:

Built-in 200 Gbps ConnectX-7

• iperf3 -c $SERVER_IP -P 4: 106 Gbps
• iperf3 -c $SERVER_IP --reverse -P 4: 106 Gbps
• iperf3 -c $SERVER_IP --bidir -P 10: 86.3 Gbps up, 86.3 Gbps down

Built-in 10 Gbps Ethernet (TODO)

• iperf3 -c $SERVER_IP: TODO Gbps
• iperf3 -c $SERVER_IP --reverse: TODO Gbps
• iperf3 -c $SERVER_IP --bidir: TODO Gbps up, TODO Gbps down

Built-in WiFi (MediaTek 7925 WiFi 7)

• iperf3 -c $SERVER_IP: 672 Mbps
• iperf3 -c $SERVER_IP --reverse: 694 Mbps
• iperf3 -c $SERVER_IP --bidir: 498 Mbps up, 198 Mbps down

GPU

glmark2

glmark2-es2 / glmark2-es2-wayland results:

1. Install glmark2-es2 with `sudo apt install -y glmark2-es2`
2. Run `glmark2-es2` (with `DISPLAY=:0` prepended if running over SSH)
3. Replace this block of text with the results.

vkmark

vkmark results:

1. Install vkmark with `sudo apt install -y vkmark`
2. Run `vkmark` (with `DISPLAY=:0` prepended if running over SSH)
3. Replace this block of text with the results.

Note: vkmark needs to be compiled from source on Debian 12 and earlier.

GravityMark

GravityMark results:

1. Download the latest version of GravityMark: https://gravitymark.tellusim.com
2. Run `chmod +x [downloaded_filename].run`
3. Run `sudo ./[downloaded_filename].run` and press `y` to accept the terms.
4. Open the link it prints, and run the Benchmark defaults, changing to 720p resolution and 50,000 asteroids.

Note: These benchmarks require an active display on the device. Not all devices may be able to run glmark2-es2, so in that case, make a note and move on!

AI / LLM Inference

See: geerlingguy/ai-benchmarks#34

Memory

tinymembench results:

Click to expand memory benchmark result

tinymembench v0.4.10 (simple benchmark for memory throughput and latency)

==========================================================================
== Memory bandwidth tests                                               ==
==                                                                      ==
== Note 1: 1MB = 1000000 bytes                                          ==
== Note 2: Results for 'copy' tests show how many bytes can be          ==
==         copied per second (adding together read and writen           ==
==         bytes would have provided twice higher numbers)              ==
== Note 3: 2-pass copy means that we are using a small temporary buffer ==
==         to first fetch data into it, and only then write it to the   ==
==         destination (source -> L1 cache, L1 cache -> destination)    ==
== Note 4: If sample standard deviation exceeds 0.1%, it is shown in    ==
==         brackets                                                     ==
==========================================================================

 C copy backwards                                     :  31501.5 MB/s (2.2%)
 C copy backwards (32 byte blocks)                    :  29732.9 MB/s
 C copy backwards (64 byte blocks)                    :  29696.7 MB/s
 C copy                                               :  31632.6 MB/s
 C copy prefetched (32 bytes step)                    :  31485.7 MB/s
 C copy prefetched (64 bytes step)                    :  31360.6 MB/s (1.2%)
 C 2-pass copy                                        :  20479.7 MB/s (0.1%)
 C 2-pass copy prefetched (32 bytes step)             :  15752.1 MB/s (0.5%)
 C 2-pass copy prefetched (64 bytes step)             :  15354.2 MB/s (0.5%)
 C fill                                               :  61894.1 MB/s
 C fill (shuffle within 16 byte blocks)               :  61766.8 MB/s
 C fill (shuffle within 32 byte blocks)               :  61767.3 MB/s
 C fill (shuffle within 64 byte blocks)               :  61793.8 MB/s
 NEON 64x2 COPY                                       :  31847.2 MB/s (0.1%)
 NEON 64x2x4 COPY                                     :  31939.7 MB/s
 NEON 64x1x4_x2 COPY                                  :  31267.7 MB/s
 NEON 64x2 COPY prefetch x2                           :  31126.2 MB/s (0.2%)
 NEON 64x2x4 COPY prefetch x1                         :  31222.3 MB/s
 NEON 64x2 COPY prefetch x1                           :  31200.8 MB/s
 NEON 64x2x4 COPY prefetch x1                         :  31209.5 MB/s
 ---
 standard memcpy                                      :  31937.3 MB/s
 standard memset                                      :  86858.0 MB/s (1.6%)
 ---
 NEON LDP/STP copy                                    :  31997.1 MB/s (0.2%)
 NEON LDP/STP copy pldl2strm (32 bytes step)          :  31861.1 MB/s
 NEON LDP/STP copy pldl2strm (64 bytes step)          :  32014.9 MB/s
 NEON LDP/STP copy pldl1keep (32 bytes step)          :  31954.3 MB/s
 NEON LDP/STP copy pldl1keep (64 bytes step)          :  31937.3 MB/s
 NEON LD1/ST1 copy                                    :  31983.2 MB/s
 NEON STP fill                                        :  86568.9 MB/s (5.1%)
 NEON STNP fill                                       :  80447.6 MB/s (3.8%)
 ARM LDP/STP copy                                     :  30950.4 MB/s (0.1%)
 ARM STP fill                                         :  86267.4 MB/s (4.7%)
 ARM STNP fill                                        :  77261.5 MB/s (1.1%)

==========================================================================
== Memory latency test                                                  ==
==                                                                      ==
== Average time is measured for random memory accesses in the buffers   ==
== of different sizes. The larger is the buffer, the more significant   ==
== are relative contributions of TLB, L1/L2 cache misses and SDRAM      ==
== accesses. For extremely large buffer sizes we are expecting to see   ==
== page table walk with several requests to SDRAM for almost every      ==
== memory access (though 64MiB is not nearly large enough to experience ==
== this effect to its fullest).                                         ==
==                                                                      ==
== Note 1: All the numbers are representing extra time, which needs to  ==
==         be added to L1 cache latency. The cycle timings for L1 cache ==
==         latency can be usually found in the processor documentation. ==
== Note 2: Dual random read means that we are simultaneously performing ==
==         two independent memory accesses at a time. In the case if    ==
==         the memory subsystem can't handle multiple outstanding       ==
==         requests, dual random read has the same timings as two       ==
==         single reads performed one after another.                    ==
==========================================================================

block size : single random read / dual random read, [MADV_NOHUGEPAGE]
      1024 :    0.0 ns          /     0.0 ns 
      2048 :    0.0 ns          /     0.0 ns 
      4096 :    0.0 ns          /     0.0 ns 
      8192 :    0.0 ns          /     0.0 ns 
     16384 :    0.0 ns          /     0.0 ns 
     32768 :    0.0 ns          /     0.0 ns 
     65536 :    0.0 ns          /     0.0 ns 
    131072 :    0.9 ns          /     1.3 ns 
    262144 :    1.3 ns          /     1.6 ns 
    524288 :    2.0 ns          /     2.3 ns 
   1048576 :    2.6 ns          /     3.0 ns 
   2097152 :    5.6 ns          /     7.2 ns 
   4194304 :   10.2 ns          /    13.3 ns 
   8388608 :   14.2 ns          /    16.1 ns 
  16777216 :   26.1 ns          /    35.0 ns 
  33554432 :   66.2 ns          /    91.1 ns 
  67108864 :   91.9 ns          /   112.4 ns 

block size : single random read / dual random read, [MADV_HUGEPAGE]
      1024 :    0.0 ns          /     0.0 ns 
      2048 :    0.0 ns          /     0.0 ns 
      4096 :    0.0 ns          /     0.0 ns 
      8192 :    0.0 ns          /     0.0 ns 
     16384 :    0.0 ns          /     0.0 ns 
     32768 :    0.0 ns          /     0.0 ns 
     65536 :    0.0 ns          /     0.0 ns 
    131072 :    0.9 ns          /     1.3 ns 
    262144 :    1.3 ns          /     1.6 ns 
    524288 :    1.6 ns          /     1.7 ns 
   1048576 :    1.7 ns          /     1.8 ns 
   2097152 :    1.9 ns          /     1.9 ns 
   4194304 :    8.5 ns          /    11.8 ns 
   8388608 :   11.6 ns          /    14.5 ns 
  16777216 :   13.4 ns          /    15.8 ns 
  33554432 :   60.3 ns          /    85.3 ns 
  67108864 :   86.3 ns          /   106.7 ns 

Core to Core Memory Latency

sbc-bench results

Run sbc-bench and paste a link to the results here: ThomasKaiser/sbc-bench#127

Phoronix Test Suite

Results from pi-general-benchmark.sh:

• pts/encode-mp3: 4.037 sec
• pts/x264 1080p: 203.94 fps
• pts/x264 4K: 52.84 fps
• pts/phpbench: 1300604
• pts/build-linux-kernel (defconfig): 188.667 sec

[geerlingguy]

Idle power draw (31.5W) seems high, with nothing but WiFi going, and a little RF dongle for keyboard and mouse (only using a few hundred mW of power at most):

The first part of the graph is before I logged in locally, with no monitor connected, and only that RF dongle.

Someone on the NVIDIA forums posted Suggestions for reducing idle power consumption?, and in there, it sounds like 30-40W idle power draw is expected (depending on what you have plugged in).

I know adding in QSFP modules will consume even more power, but that's why I had them disconnected.

I'm a little disappointed there, but maybe it comes from adapting an APU architecture meant for servers down to a mini PC. It is certainly silent, still, and doesn't feel too hot, even after running an hour or two. I did observe clocks down to the 2.2 GHz range or so on the CPU side, and nvtop showed the GPU as running at PCIe Gen 1x1 at idle.

[geerlingguy]

Also, I haven't started testing the ConnectX-7 ports yet (see guide Connect Two Sparks), but apparently you'll get somewhere around 200 Gbps maximum across both ports, you can't do 200 + 200 Gbps bonded to get 400 Gbps between machines.

See: https://forums.developer.nvidia.com/t/connectx-7-nic-in-dgx-spark/350417/20

Despite your optimistic LLM telling you what you want to hear, you won’t be able to achieve 400G using two ports - when both ports are active, each will get 100G. DGX Spark doesn’t have enough PCI lanes to achieve 400G, and even to achieve 200G on a single port, they had to do some voodoo with dual NIC and bonding due to PCIe limitations on this arch.

From elsewhere in that thread, and from talking to Patrick from STH, I discovered each port gets a PCIe Gen 5 x4 connection to the CPU. And a trip to lspci confirms that:

0002:01:00.1 Ethernet controller: Mellanox Technologies MT2910 Family [ConnectX-7]
	Subsystem: Mellanox Technologies MT2910 Family [ConnectX-7]
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin B routed to IRQ 468
	IOMMU group: 19
	Region 0: Memory at 3d02000000 (64-bit, prefetchable) [size=32M]
	Expansion ROM at 5d200000 [virtual] [disabled] [size=1M]
	Capabilities: [60] Express (v2) Endpoint, MSI 00
		DevCap:	MaxPayload 512 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0W
		DevCtl:	CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
			RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
			MaxPayload 512 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 32GT/s, Width x4, ASPM not supported
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 32GT/s, Width x4
			TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-

[geerlingguy]

Two times I've tried running Geekbench 6 on this system, and two times it seems like it stalls out on the Object Remover multicore test. The power consumption spikes and stays there, and the Geekbench thread just sits. I can Ctrl-C out of it without issue, but I can't get a full Geekbench 6 run to complete.

Someone did get one successful run in during September: https://browser.geekbench.com/v6/cpu/14078585 (3237 single / 19794 multi), but I also noticed other runs were odd in the multicore speedup or single core speeds, so I wonder if they were messing with clocks, memory, or performance profiles, to make it run successfully?

[geerlingguy]

After four attempts, I finally got a result: https://browser.geekbench.com/v6/cpu/15161960

[geerlingguy]

I've been considering testing latest Fedora/Ubuntu on the box, since it has UEFI support... found the following issues:

• Has anyone tried an alternative Linux distro?
• Ubuntu 26.04 LTS (Kernel: 6.17.0) ARM64 on DGX Spark, anyone?

Looks like for now, if you use a later kernel/different OS, you need to manually install r8127 for 10G Ethernet, and maybe the latest Nvidia open drivers for graphical support. Some things work, some things don't, some of that is documented in the above issues.

[eugr]

If you manage to get close to 200G between two Sparks, as measured by ib_write_bw or iperf3, please let me know. I tried to create a bond like one of the Nvidia engineers suggested, but haven't been able to make it work so far. Not in xor mode, not in LACP (802.3ad).

As for an alternative OS, Fedora 43 with custom Nvidia kernel compiled from their repo works pretty well. The one that comes with Fedora works after installing nvidia drivers, but you need to install r8127 separately (haven't tried that outside of Nvidia kernel) and the GPU performance with Fedora kernel was slower, but the last one I tried was 6.17.5, maybe the newest one would work better.

I've switched back to stock DGX OS for now, as they improved model loading speed on their 6.14 kernel, but mmap performance on any kernel remains mediocre and borderline unusable (I mean, waiting 8 minutes for vllm to load Qwen3-Next-80B?!!!). I hope this is not a hardware limitation, but there was a post from one of their engineers explaining why RDMA to GPU direct doesn't work, and it turns out that once CUDA allocates memory pool for GPU, it becomes isolated from CPU and PCIe, so I suspect it means that you can't do NVME->VRAM mapping because of that. If so, that's a major flaw that nobody is talking about.

[geerlingguy]

@eugr - I believe it may have been Patrick from STH who mentioned using only the left ports to connect node to node, to get 200 Gbps performance? There was some weird architecture quirk he was hinting at, but this was just talking about it in passing over a short walk at SC25 last week, so I may have missed something.

Haven't yet plugged the two boxes together, but we'll see!

[eugr]

I'm using the outer port which is on the left if looking from the front. I can try the other one to see if there is any difference.
What's weird is that when I run NCCL tests it shows bus bandwidth as 22 GB/s which is about right for 200G.

[geerlingguy]

Some CX7 Ethernet debugging...

I have the cable plugged between the left-most ports on the back of the devices:

There are two ConnectX-7 devices showing up on the PCIe bus (with two entries each, per port):

jgeerling@dell-gb10-1:~/nccl-tests$ lspci
0000:00:00.0 PCI bridge: NVIDIA Corporation Device 22ce (rev 01)
0000:01:00.0 Ethernet controller: Mellanox Technologies MT2910 Family [ConnectX-7]
0000:01:00.1 Ethernet controller: Mellanox Technologies MT2910 Family [ConnectX-7]
0002:00:00.0 PCI bridge: NVIDIA Corporation Device 22ce (rev 01)
0002:01:00.0 Ethernet controller: Mellanox Technologies MT2910 Family [ConnectX-7]
0002:01:00.1 Ethernet controller: Mellanox Technologies MT2910 Family [ConnectX-7]
...

# And showing the Link info for one of the two:
jgeerling@dell-gb10-1:~/nccl-tests$ sudo lspci -s 0000:01:00 -vvv
0000:01:00.0 Ethernet controller: Mellanox Technologies MT2910 Family [ConnectX-7]
	Subsystem: Mellanox Technologies MT2910 Family [ConnectX-7]
...
		LnkCap:	Port #0, Speed 32GT/s, Width x4, ASPM not supported
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 32GT/s, Width x4
			TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-

I followed Nvidia's guide for DGX Spark cluster networking and set up the links like:

jgeerling@dell-gb10-1:~/Downloads/top500-benchmark$ sudo cat /etc/netplan/40-cx7.yaml
network:
  version: 2
  ethernets:
    enp1s0f0np0:
      link-local: [ ipv4 ]
    enp1s0f1np1:
      link-local: [ ipv4 ]

(with sudo chmod 600 /etc/netplan/40-cx7.yaml and sudo netplan apply).

I can ping each node from the other:

jgeerling@dell-gb10-1:~$ ping 169.254.98.29
PING 169.254.98.29 (169.254.98.29) 56(84) bytes of data.
64 bytes from 169.254.98.29: icmp_seq=1 ttl=64 time=0.805 ms
...
jgeerling@dell-gb10-2:~$ ping 169.254.233.24
PING 169.254.233.24 (169.254.233.24) 56(84) bytes of data.
64 bytes from 169.254.233.24: icmp_seq=1 ttl=64 time=0.838 ms

Testing with iperf3, I get:

jgeerling@dell-gb10-1:~$ iperf3 -c 169.254.98.29
...
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  34.6 GBytes  29.7 Gbits/sec  9721             sender
[  5]   0.00-10.00  sec  34.6 GBytes  29.7 Gbits/sec                  receiver

If I use multiple parallel streams (-P 4), I can achieve around 100 Gbps (thanks to Patrick from STH for the suggestion):

jgeerling@dell-gb10-1:~$ iperf3 -c 169.254.98.29 -P 4
Connecting to host 169.254.98.29, port 5201
[  5] local 169.254.233.24 port 51168 connected to 169.254.98.29 port 5201
[  7] local 169.254.233.24 port 51174 connected to 169.254.98.29 port 5201
[  9] local 169.254.233.24 port 51182 connected to 169.254.98.29 port 5201
[ 11] local 169.254.233.24 port 51188 connected to 169.254.98.29 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  3.07 GBytes  26.4 Gbits/sec  667   1.07 MBytes       
[  7]   0.00-1.00   sec  2.91 GBytes  25.0 Gbits/sec  1017   1.55 MBytes       
[  9]   0.00-1.00   sec  3.39 GBytes  29.1 Gbits/sec  512   1.08 MBytes       
[ 11]   0.00-1.00   sec  2.90 GBytes  24.9 Gbits/sec  796   1.12 MBytes       
[SUM]   0.00-1.00   sec  12.3 GBytes   105 Gbits/sec  2992             
- - - - - - - - - - - - - - - - - - - - - - - - -
...
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  30.2 GBytes  26.0 Gbits/sec  4355             sender
[  5]   0.00-10.00  sec  30.2 GBytes  26.0 Gbits/sec                  receiver
[  7]   0.00-10.00  sec  29.2 GBytes  25.1 Gbits/sec  1625             sender
[  7]   0.00-10.00  sec  29.2 GBytes  25.1 Gbits/sec                  receiver
[  9]   0.00-10.00  sec  34.2 GBytes  29.4 Gbits/sec  5392             sender
[  9]   0.00-10.00  sec  34.2 GBytes  29.4 Gbits/sec                  receiver
[ 11]   0.00-10.00  sec  29.3 GBytes  25.2 Gbits/sec  5328             sender
[ 11]   0.00-10.00  sec  29.3 GBytes  25.2 Gbits/sec                  receiver
[SUM]   0.00-10.00  sec   123 GBytes   106 Gbits/sec  16700             sender
[SUM]   0.00-10.00  sec   123 GBytes   106 Gbits/sec                  receiver

And ethtool is showing the port attaching at 200 Gbps:

jgeerling@dell-gb10-1:~$ ethtool enp1s0f0np0
Settings for enp1s0f0np0:
	Supported ports: [ Backplane ]
...
	Link partner advertised link modes:  Not reported
	Link partner advertised pause frame use: No
	Link partner advertised auto-negotiation: Yes
	Link partner advertised FEC modes: Not reported
	Speed: 200000Mb/s
	Duplex: Full
	Auto-negotiation: on
	Port: Direct Attach Copper
	PHYAD: 0
	Transceiver: internal
netlink error: Operation not permitted
	Link detected: yes

NCCL Tests

Nvidia recommends testing cluster networking via NCCL:

Running their test, I got 22.937 Gbps of bandwidth...

jgeerling@dell-gb10-1:~/nccl-tests$ mpirun -np 2 -H 169.254.233.24:1,169.254.98.29:1   --mca plm_rsh_agent "ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no"   -x LD_LIBRARY_PATH=$LD_LIBRARY_PATH   $HOME/nccl-tests/build/all_gather_perf -b 16G -e 16G -f 2
Authorization required, but no authorization protocol specified

Authorization required, but no authorization protocol specified

Warning: Permanently added '169.254.98.29' (ED25519) to the list of known hosts.
Authorization required, but no authorization protocol specified

# nccl-tests version 2.17.6 nccl-headers=22803 nccl-library=22803
# Collective test starting: all_gather_perf
# nThread 1 nGpus 1 minBytes 17179869184 maxBytes 17179869184 step: 2(factor) warmup iters: 1 iters: 20 agg iters: 1 validation: 1 graph: 0
#
# Using devices
#  Rank  0 Group  0 Pid  31385 on dell-gb10-1 device  0 [000f:01:00] NVIDIA GB10
#  Rank  1 Group  0 Pid 130438 on dell-gb10-2 device  0 [000f:01:00] NVIDIA GB10
#
#                                                              out-of-place                       in-place          
#       size         count      type   redop    root     time   algbw   busbw  #wrong     time   algbw   busbw  #wrong 
#        (B)    (elements)                               (us)  (GB/s)  (GB/s)             (us)  (GB/s)  (GB/s)         
 17179869184    2147483648     float    none      -1   374595   45.86   22.93       0   374408   45.89   22.94       0
# Out of bounds values : 0 OK
# Avg bus bandwidth    : 22.937 
#
# Collective test concluded: all_gather_perf

Hrm...

[geerlingguy]

RDMA / Infiniband testing

jgeerling@dell-gb10-1:~$ sudo apt install infiniband-diags
jgeerling@dell-gb10-1:~$ ibstat
CA 'roceP2p1s0f0'
	CA type: MT4129
	Number of ports: 1
	Firmware version: 28.45.4028
	Hardware version: 0
	Node GUID: 0xfc4cea0300f94234
	System image GUID: 0xfc4cea0300f94230
	Port 1:
		State: Active
		Physical state: LinkUp
		Rate: 200
		Base lid: 0
		LMC: 0
		SM lid: 0
		Capability mask: 0x00010000
		Port GUID: 0xfe4ceafffef94234
		Link layer: Ethernet
CA 'roceP2p1s0f1'
	CA type: MT4129
	Number of ports: 1
	Firmware version: 28.45.4028
	Hardware version: 0
	Node GUID: 0xfc4cea0300f94235
	System image GUID: 0xfc4cea0300f94230
	Port 1:
		State: Down
		Physical state: Disabled
		Rate: 40
		Base lid: 0
		LMC: 0
		SM lid: 0
		Capability mask: 0x00010000
		Port GUID: 0xfe4ceafffef94235
		Link layer: Ethernet
CA 'rocep1s0f0'
	CA type: MT4129
	Number of ports: 1
	Firmware version: 28.45.4028
	Hardware version: 0
	Node GUID: 0xfc4cea0300f94230
	System image GUID: 0xfc4cea0300f94230
	Port 1:
		State: Active
		Physical state: LinkUp
		Rate: 200
		Base lid: 0
		LMC: 0
		SM lid: 0
		Capability mask: 0x00010000
		Port GUID: 0xfe4ceafffef94230
		Link layer: Ethernet
CA 'rocep1s0f1'
	CA type: MT4129
	Number of ports: 1
	Firmware version: 28.45.4028
	Hardware version: 0
	Node GUID: 0xfc4cea0300f94231
	System image GUID: 0xfc4cea0300f94230
	Port 1:
		State: Down
		Physical state: Disabled
		Rate: 40
		Base lid: 0
		LMC: 0
		SM lid: 0
		Capability mask: 0x00010000
		Port GUID: 0xfe4ceafffef94231
		Link layer: Ethernet

Then to perform a bandwidth test:

# ON HOST:
jgeerling@dell-gb10-2:~$ ib_send_bw
 WARNING: BW peak won't be measured in this run.

************************************
* Waiting for client to connect... *
************************************
...

# ON CLIENT:
jgeerling@dell-gb10-1:~$ sudo ib_send_bw 169.254.98.29 -d rocep1s0f0 --report_gbits
---------------------------------------------------------------------------------------
                    Send BW Test
 Dual-port       : OFF		Device         : rocep1s0f0
 Number of qps   : 1		Transport type : IB
 Connection type : RC		Using SRQ      : OFF
 PCIe relax order: ON
 ibv_wr* API     : ON
 TX depth        : 128
 CQ Moderation   : 1
 Mtu             : 1024[B]
 Link type       : Ethernet
 GID index       : 3
 Max inline data : 0[B]
 rdma_cm QPs	 : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x0171 PSN 0x6813eb
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:169:254:233:24
 remote address: LID 0000 QPN 0x0171 PSN 0x8985b
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:169:254:98:29
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[Gb/sec]    BW average[Gb/sec]   MsgRate[Mpps]
 65536      1000             108.84             108.82 		   0.207564
---------------------------------------------------------------------------------------

[raphaelamorim]

Have you tried multiple instances of iperf3 running at -P 4 to check if it's really using the multiple links simultaneously?