Microsoft Lets Multiple PCs Run One LLM Over USB4

Microsoft has released a developer preview of RDMA over USB4 for Windows. Several Windows PCs connected with ordinary USB4 cables can pool their memory and GPUs to run a single local large language model, with no dedicated RDMA network cards, no switch and no third-party drivers.

The feature targets models too big for one machine's memory. Until now the options were to shrink the model further or hand the request to the cloud. Now there is a third: plug in another PC.

Illustration: three mini PCs linked pairwise by three cables support a red cube representing one shared local AI model

Three mini PCs running a 235B model

Microsoft published two sets of demos, one on AMD hardware and one on Intel.

SetupModelGeneration speed
3 x Ryzen AI Max+ 395, 128GB eachQwen3-235B-A22B (4-bit quantized)18.5 to 21.98 tokens/s
3 x Core Ultra Series 3, 64GB eachGPT-OSS-120B49.4 tokens/s
4 x Core Ultra Series 3, 64GB eachGPT-OSS-120B54.2 tokens/s

The AMD setup uses mini PCs with Radeon 8060S integrated graphics and 128GB of unified memory each. The 18.5 figure was measured by Microsoft; the 21.98 figure comes from AMD running its own configuration. In the Intel demo, going from three machines to four lifted speed from 49.4 to 54.2 tokens per second, so the extra machine bought less than a 10% gain.

A back-of-the-envelope calculation shows why pooling is needed. At 4-bit precision, 235 billion parameters take close to 118GB for the weights alone. On a single 128GB machine that leaves almost no headroom for context and the operating system; spread across three, there is room for long contexts. This is an estimate based on parameter count. Actual usage depends on the quantization format and context length, and Microsoft has not published per-node memory usage.

Speed comes from bypassing the network stack

RDMA, or remote direct memory access, lets data move between two machines while skipping most of the regular networking software. Microsoft puts round-trip latency for small messages at roughly 28 to 50 microseconds, about four times faster than UDP over the same USB4 link. Link speeds of 40Gbps and 80Gbps are supported, and compatible Thunderbolt 4 and Thunderbolt 5 ports also work.

Distributed inference is very sensitive to latency. When a model is split across machines, intermediate results must make a round trip between them for every token generated, so any slowdown in transfers drags down overall speed.

Cabling matters. Microsoft requires every PC to connect directly to every other PC: with three machines, each uses two USB4 ports and the setup needs three cables. Extending that rule, four machines would need three ports each and six cables. Microsoft specifically warns that daisy-chaining machines noticeably hurts performance. Consumer PCs have only a few USB4 ports, so we estimate the practical ceiling for this approach at around four or five machines.

How it compares with dedicated NIC setups

RDMA has long been used in servers and AI clusters, and both Windows and Linux have supported it for years, but it has always required dedicated hardware. ServeTheHome previously demonstrated an eight-node Nvidia GB10 cluster with 1TB of combined memory, using ConnectX-7 network cards and 200Gbps Ethernet over the RoCE protocol. That setup offers several times the link speed of USB4, at the cost of a dedicated NIC in every machine plus a switch.

Apple took a similar route to Microsoft's. Late last year it opened up RDMA over Thunderbolt 5 in macOS, and stacking several Macs to run large models became more common afterward. On the Windows side, VideoCardz noted a detail: RTX Spark PCs lack the ConnectX-7 port found on the DGX Spark, and Microsoft's feature is aimed squarely at that kind of machine, as well as AMD's Strix Halo and Gorgon Halo mini PCs.

That makes the position of the USB4 approach clear. It cannot match data-center setups on bandwidth or scalability, but it needs no extra hardware, so small teams and individuals with two or three recent PCs can try it.

When can you use it

The feature is not in any publicly available Windows build yet. Microsoft plans to ship it through the Experimental channel of the Windows Insider program, with no date disclosed. The underlying NetworkDirect provider and a modified build of the llama.cpp inference framework are already open source, so developers who want to get started early can look at the code.

Microsoft has also left several questions unanswered: which USB4 controller chips are on the supported list, whether machines from different brands can be mixed, and when popular tools such as Ollama and LM Studio will follow. The demos all used new-generation chips, and whether older USB4 PCs will work will only be clear from real-world tests once the preview ships.

Sources: Microsoft USB tech blog, VideoCardz, CocoLoop; we checked the machine configurations, models and tokens-per-second figures for the three demos, as well as the round-trip latency and cabling requirements.