Meta published design details for MetaRoCE on August 24. It is an in-house RDMA transport protocol built to replace standard RoCE. At the OCP Global Summit in October, Meta will release the full protocol specification, a reference implementation called libsoftmetaroce, and a production conformance test suite.
Standard RoCE Breaks Down at Hundred-Thousand-GPU Scale
RoCE assumes the network delivers every frame in order, relying on PFC (Priority Flow Control) on switches to guarantee lossless delivery. That assumption holds up fine on clusters with a few hundred cards, but it starts to leak at the scale Meta operates today.
In Meta's own words, standard RoCE expects the network to deliver every frame in order, depends on PFC, and discourages packet spraying — even though packet spraying is exactly the technique that improves performance on large multi-plane networks. That is the conflict: spraying a single flow's packets across multiple paths at once runs directly against in-order delivery. PFC is also hop-by-hop backpressure, so congestion at one point pushes back all the way down the chain, and at extreme scale that can cascade. And collective communication operations like all-reduce have to synchronize thousands of accelerators at once, so the slowest transfer sets the pace for the entire job.
The target scenario Meta describes is “hundreds of thousands of GPUs distributed across multiple data centers and regions.” At that scale, relying on switches to maintain a lossless network is already a bet against the odds.
Pushing the Intelligence to the NIC
MetaRoCE's design principle is distilled into one line:
“The fabric sees packets, but the NIC sees intent.”
That becomes six concrete features: native out-of-order delivery, spraying packets across multiple paths on purpose and writing data straight to the destination memory address with no reorder buffer; native multipathing, where each connection maintains multiple logical paths that each track their own round-trip latency and ECN state; loss tolerance, treating packet loss as normal and patching gaps with selective-acknowledgment bitmaps instead of needing PFC; dual-loop congestion control, combining sender-side AIMD with receiver-side fair-share rate hints; topology independence, requiring only ECN marking and ECMP so it runs on fat-tree, multi-plane, or shallow-buffer switches alike; and unified connections, where a single queue carries multiple ordered message streams and multiple paths sharing one congestion controller.
The Numbers on a 64-Node Cluster
Meta tested MetaRoCE on a 64-node AMD GPU cluster. Throughput stayed consistently above RoCEv2; at 1% packet loss it held onto about 86% of throughput, and it kept working even at 10% loss; throughput scaled linearly across 4-plane and 8-plane topologies, tested up to 4,000 concurrent connections; and when a single plane failed, traffic rerouted itself without any application-layer intervention.
There is still a long way from 64 nodes to “hundreds of thousands of cards,” and Meta breaks the remaining work into three buckets: scale-up within a rack needs faster signaling for short memory operations to strip out the latency added by reorder buffers and PFC; scale-across links spanning thousands of kilometers needs per-path adaptation and fairness tuned for millisecond-scale round trips; and storage and KV-cache scenarios plan to use receiver-side rate hints to handle incast, where a single read fans out across many servers at once.
Open-Sourcing It Is Aimed at the NIC Vendors
On the NIC side, AMD Pensando is currently the only verified implementation, with others reportedly in progress. That single fact will decide how far MetaRoCE actually goes — no matter how elegant the protocol is, if it can't land in mainstream NIC firmware, it stays an internal Meta project.
Releasing it through OCP and explicitly aligning with the ESUN (Ethernet for Scale-Up Networking) initiative makes the intent obvious: get NIC vendors shipping to this spec so Meta's own supplier base widens. Ethernet has run this play before — RoCE originally clawed share away from InfiniBand precisely because the Ethernet ecosystem was cheaper and had more vendors. Now that RoCE itself has become the bottleneck, Meta wants to run the same playbook again: put the spec on the table first, and let everyone else build to it.
For teams building AI compute infrastructure in China, the full specification due in October is worth a close read, especially the “no PFC required” part. Cutting PFC tuning out of the picture removes one of the thorniest sources of operational pain.
Sources: Meta Engineering blog, CocoLoop, OCP ESUN materials; figures on ~86% throughput at 1% loss, continued operation at 10% loss, the 64-node AMD GPU cluster, and 4,000 concurrent connections are all cross-checked against Meta's published test results.