📝 gpu_network_protocols.mdTheme:

The Named Protocols That Connect GPUs, From the Model Down to the Wire

This document lists every named technology that carries data between GPUs, working from the language model at the top to the cables at the bottom. For each one it says what it connects and how it differs from the others. The last section covers how the networks differ between training (building the model) and inference (using it to answer people).

Sources are tagged: [T] appears in the transcript, [K] comes from my own background knowledge, [W] comes from a web search I ran today.

1. The whole stack at a glance

Level	Names	What it connects	Hardware or software
Model	Neural network, layers, attention, feed-forward network [T]	Words and numbers to layers	Software
Ways of splitting the model	Data, tensor, pipeline and expert parallelism [K]	Decides which GPU holds what, and so who must talk to whom	Software
Group-communication routines	All-reduce [T], all-gather, all-to-all, NCCL [K]	A group of GPUs exchanging results together	Software
Inside one server	PCIe, HBM, NVLink, NVSwitch, Infinity Fabric [K]	A GPU to its memory, its server, and its closest neighbours	Hardware
Between servers	InfiniBand, Ethernet, RoCE v2 [T/K]	Servers across the cluster	Hardware
Ways of sending over it	RDMA [T], UDP, IP, MRC, Ultra Ethernet [T/K/W]	Rules for moving the data without jams or loss	Mostly network-card and switch firmware
Bridge between the two	GPUDirect RDMA [K]	A network card straight to GPU memory	Hardware and driver

2. How the model is split (and why that decides the traffic)

A model too large for one GPU has to be divided. The four common ways are named, and each creates a different kind of network traffic. [K]

Data parallelism: every GPU holds a full copy of the model and works on different data. After each step they average their gradients with all-reduce, the exchange the transcript describes. This is the heavy, whole-cluster traffic.
Tensor parallelism: one layer's calculation is split across several GPUs. They must exchange results many times per layer, so this only works over the fastest links, inside one server.
Pipeline parallelism: different layers sit on different GPUs, and results pass from one stage to the next. The traffic is lighter and can cross servers.
Expert parallelism: used by "mixture of experts" models, which send each word to a few specialist sub-networks living on different GPUs. This creates an all-to-all pattern, where every GPU sends to many others.

3. The group-communication routines

Training does not send single messages. It runs group operations: all-reduce (average everyone's numbers and hand the result back), all-gather (everyone receives everyone's pieces), and all-to-all (everyone sends something different to everyone else). NCCL (NVIDIA Collective Communications Library, said "nickel") is the software that carries these out, choosing the best route among the links below. AMD has an equivalent called RCCL. [K]

4. Inside the box: connections within one server or rack

Name	What it connects	How it differs
HBM (high-bandwidth memory)	A GPU to its own memory, stacked on the same package	Not a network, but the fastest data path in the system
PCIe (PCI Express)	The GPU to the server's processor and network cards	General-purpose and slower than the GPU-to-GPU links
NVLink	GPU to GPU, within NVIDIA systems	Far faster than PCIe and not IP-based
NVSwitch	Many GPUs through one switch chip, so each can reach all the others at full speed	Makes a whole server, or some racks of 72 GPUs, behave like one big GPU
Infinity Fabric	GPU to GPU in AMD systems	AMD's equivalent of NVLink
ICI (Inter-Chip Interconnect)	Google's TPU chips to each other	The same idea for non-GPU accelerators
UALink (Ultra Accelerator Link)	An open standard for the same job, backed by AMD, Intel, Broadcom and others	Intended as a vendor-neutral alternative to NVLink

Networks at this level are called scale-up. They are short, extremely fast and tightly controlled. [K]

5. Between boxes: the cluster network

This is the network the transcript is about, called scale-out. [T]

The two physical choices

InfiniBand: a network designed from scratch for high-performance computing, with its own addressing and routing and no IP. Long dominant in AI clusters, and mostly supplied by NVIDIA. [K]
Ethernet: the same family as home and office networks, made much faster. Cheaper, more widely supplied, and now the main rival to InfiniBand. [K]

The ways of sending data across them

Name	What it is	How it differs
RDMA (Remote Direct Memory Access) [T]	The sending network card writes directly into another machine's memory	Skips the receiver's processor and operating system
RoCE v2 (RDMA over Converged Ethernet) [T/K]	RDMA carried inside UDP and IP packets on Ethernet	Uses IP, so it routes across ordinary Ethernet switches
iWARP [K]	RDMA carried over TCP	An older alternative, rarely used for AI because TCP adds delay
UDP, IP, TCP [K]	The ordinary internet protocols	RoCE v2 uses UDP and IP as an addressing envelope. TCP is too heavy for the fast path and is kept for setup and housekeeping
Ultra Ethernet [K]	An open set of standards for AI traffic on Ethernet, from an industry consortium	Designed for the weaknesses of plain RoCE
MRC (Multipath Reliable Connection) [W]	A transport that extends RoCE v2 for very large clusters	See section 6

Helper mechanisms

PFC (Priority Flow Control) tells a sender to pause. ECN (Explicit Congestion Notification) lets switches flag congestion early. Plain RoCE depends on these to avoid loss. [K]
SRv6 (Segment Routing over IPv6) lets the sender write the route into the packet itself. [W]

6. The transcript's mystery name is probably MRC

The transcript's "Metrosy" matches MRC in both sound and description, so I'm treating it as MRC. That is my inference, since the speaker's exact word is unclear. What I found: [W]

It was developed by OpenAI together with Microsoft, NVIDIA, AMD, Intel and Broadcom, unveiled in May 2026, and published as an open specification through the Open Compute Project.
It extends RoCE v2 rather than replacing it. Endpoints using MRC can't talk to plain RoCE endpoints.
It sprays packets: each packet can take a different path, instead of one connection being pinned to one path.
It resends only what is lost: the receiver reports exactly which packets arrived, so only the missing ones are sent again. Switches can also shrink a packet to just its header when overloaded, so the receiver knows at once what to request.
It watches for congestion from both ends, using ideas from Ultra Ethernet.
It does not depend on PFC pausing, and it can route around failed links quickly using SRv6.
OpenAI reports it is already running on its largest training systems.

7. The bridge: GPUDirect RDMA

Without it, data leaving a GPU would first be copied into the server's main memory, then handed to the network card. GPUDirect RDMA lets the network card read and write GPU memory directly. It is what joins the fast links inside the box to the cluster network between boxes. [K]

8. Training versus inference

Training builds the model. Inference is the model answering a request, for example turning your question into a reply. The networks behave quite differently.

	Training	Inference
Goal	Finish a job lasting weeks or months	Answer each request quickly
What matters most	Total throughput across the whole cluster	Delay on every single response
Traffic pattern	All GPUs in lockstep, sharing numbers after every step (all-reduce)	Many independent, small requests
GPUs involved	Thousands working as one machine	Often one server, or a few, per copy of the model
Main network	Cluster network (scale-out), with RDMA	The links inside a server (scale-up) first
Cost of one slow link	Stalls every GPU in the job	Slows one response
Touches the ordinary internet?	No	Yes, at the front door

Three more differences matter:

Where the internet comes in: a user's request arrives over ordinary internet traffic (HTTP over TCP) at a front-end server, which hands it to the GPUs. That stage uses the plain protocols. Training has no such stage. [K]
Two phases of inference: reading your prompt (prefill) is heavy on calculation. Producing the reply word by word (decode) is limited by memory speed, and it demands very low delay from the tensor-parallel links inside the server. [K]
The cache that travels: large services often put prefill and decode on separate groups of GPUs. The model's working memory of the conversation so far, called the KV cache (key-value cache), must then be sent between them. That is RDMA traffic between servers, and software such as NVIDIA's NIXL and the open Mooncake engine exists for it. Very large mixture-of-experts models also send all-to-all traffic across servers during inference. [K]

So inference needs the cluster network much less than training, but it still needs it for the biggest models and for moving the cache.

9. What each network does differently, in one table

Network	Connects	Uses IP?	Typical use
HBM	GPU to its own memory	No	Every calculation
PCIe	GPU to server processor and network card	No	Setup, data loading
NVLink / NVSwitch / Infinity Fabric	GPU to GPU inside a server or rack	No	Tensor parallelism; training and inference
InfiniBand	Servers to servers	No	Training clusters
Ethernet with RoCE v2	Servers to servers	Yes (UDP and IP)	Training clusters; large inference
Ethernet with MRC	Servers to servers at very large scale	Yes	Largest training clusters
Ordinary internet (TCP)	Users to the service	Yes	Inference front door; training housekeeping