The Different "Networks" in AI Training: A Map

The word "network" is used for several unrelated things in this topic. Some are software and some are hardware. They sit at different levels, and mixing them up is the main source of confusion.

The short version

#	What it's called	What it really is	Software or hardware	What it does
1	Neural network	The AI model itself: layers of numbers (called weights) that transform input into output	Software (mathematics)	An LLM is a neural network
2	Feed-forward network (FFN)	One block of calculations inside each layer of the model	Software (mathematics)	Transforms each word's numbers and passes them on
3	Links inside one server or rack	Very fast direct connections between GPUs that sit close together (NVIDIA's NVLink is a well-known example)	Hardware	Lets a handful of GPUs behave almost like one big chip
4	The cluster network	The cables, switches and network cards (NICs) that connect all the servers, using Ethernet or InfiniBand	Hardware	Carries data between machines; this is the subject of the transcript
5	Transport methods (RDMA, RoCE, and the spraying-across-paths scheme)	Rules and tricks for how data is sent over the cluster network	Mostly built into network cards and switch software	Make that network fast, and keep it from jamming or losing data
6	The internet	The ordinary public web	Hardware and software	Not involved in the training traffic at all

Level by level

1. The neural network: the model

This is the thing being trained. It is a huge stack of layers, each holding numbers that get adjusted during training. When people say "the AI", this is what they mean. It is an idea expressed in maths, and it is not a physical object.

2. The feed-forward network: a part inside the model

Each layer of an LLM has two main parts. One is attention, which lets each word look at the other words around it. The other is the feed-forward network, which takes each word's numbers and transforms them. The FFN is a building block within the model. It has nothing to do with cables.

Layers pass their results to the next layer, so numbers flow through the model from first layer to last. That flow is part of the model's design.

3. Links inside one server or rack: the short, fast hop

When the model is too big for one chip, it is spread across GPUs. The GPUs inside the same server or rack are wired to each other with special high-speed links, much faster than ordinary network cabling. They exist because those GPUs talk to each other the most.

4. The cluster network: the long hop

A cluster is made of many servers, and these need a network too. This is the physical network the transcript is about: cables, switches, and the network cards in each machine. It is slower and shared by thousands of GPUs at once, which is where the traffic jams happen.

What crosses it:

• Gradients: the adjustment numbers each GPU shares so that all of them update the model the same way. The averaging step is called all-reduce.
• Intermediate results: numbers passed between GPUs when different layers live on different chips.

5. Transport methods: how the data is sent

These are not separate cables. They are techniques that run on the cluster network:

• RDMA (Remote Direct Memory Access): the sending network card writes straight into another machine's memory, bypassing that machine's processor and operating system.
• RoCE (RDMA over Converged Ethernet): RDMA carried over ordinary Ethernet cabling. The transcript renders it as "Rose".
• The spraying scheme: a newer method that spreads packets across many routes, resends only lost packets, and watches for congestion from both ends. The transcript garbles its name ("Metrosy"), so I can't identify it with confidence.

6. The internet: not involved

Ordinary web traffic is spread out and unpredictable. Training traffic is the opposite, with thousands of machines sending to each other in lockstep. That is why the cluster network has to be built so differently from the internet.

How the pieces fit together

Read it from the top down:

1. The neural network (1) is the model, made of layers with feed-forward networks (2) and attention inside them.
2. The model is too big for one chip, so its layers are spread across GPUs.
3. GPUs close together talk over fast local links (3). GPUs further apart talk over the cluster network (4).
4. The cluster network uses transport methods (5) to avoid jams and delay.
5. If the cluster network is slow, the chips sit idle waiting for each other's results, and the next training step can't begin.

Rows 1 and 2 are what the model computes. Rows 3 to 5 are how the results travel between the chips doing the computing.