An AI datacentre is built as a set of nested units, each one a bigger copy of the one before. This document walks up from a single chip to a whole site. At each level it says how many there are, what hardware is typically used, and how the units at that level are connected.
Tags: [W] a figure from a web source I read today (listed at the end), [K] my own background knowledge, [Calc] my arithmetic on those figures.
Reading the speeds: a byte is 8 bits. A speed written GB/s (gigabytes per second) is eight times bigger than the same number written Gbps (gigabits per second). Both are used below, and the numbers can look misleading side by side.
| Level | What it is | Typical count | What connects the units |
|---|---|---|---|
| Chip | One accelerator (a GPU, or Google's TPU, Tensor Processing Unit) | 1 | Its own stacked memory (HBM) |
| Tray or server | A few chips plus a processor, in one box | About 4 chips per tray [W] | PCIe inside the box, plus the fast links |
| Rack | The unit that now matters most: one cabinet of trays, wired into a single fast group | 72 GPUs in NVIDIA's and AMD's current racks [W] | Scale-up links (NVLink, or UALink over Ethernet) |
| Pod or row | A set of racks sold or built as one block | 8 racks, or 576 GPUs, in NVIDIA's SuperPOD design [W] | Scale-out network |
| Cluster | Everything one training job can use | Thousands to hundreds of thousands of GPUs | Scale-out network, in two or three switch tiers |
| Site | Buildings, power plant and cooling for the cluster | Several buildings of about 50,000 GPUs each [W] | Scale-out network, plus the power and cooling infrastructure |
The rack is where the fast links stop and the slower cluster network begins, which is the most important boundary in the design.
| Site | Reported size | Source quality |
|---|---|---|
| Stargate, Abilene, Texas | Up to 400,000 NVIDIA GB200 chips across eight buildings, about 50,000 per building, 1.2 GW planned | Press report from early 2025 [W] |
| xAI Colossus, Memphis | Reports of 555,000+ GPUs and about 2 GW | Secondary summary whose own numbers don't add up [W] |
| Largest Ethernet design shown | About 131,000 GPUs under only two switch tiers, using 800 Gbps cards and 512-port switches | From the MRC announcement [W] |
At 72 GPUs per rack, a 400,000-GPU site is about 5,500 racks, and a 100,000-GPU cluster is about 1,400 racks. [Calc]
| Unit | What it does | How it connects |
|---|---|---|
| Front-end network | Brings in users and data, and reaches storage | Ordinary Ethernet and the internet protocols, kept apart from the training network [W] |
| Storage | Holds training data and the saved snapshots (checkpoints) that let a job survive a failure | Its own storage network, reached via the front end [K] |
| Management network | Monitoring, control and booting | A slow, separate network [K] |
| Scheduler | Software that assigns jobs to GPUs and restarts work after a failure | Runs over the management and front-end networks [K] |
| Power | Feeds the racks | At rack level through busbars. At site level it can need its own generating plant, as the Colossus reports describe [W] |
| Cooling | Removes the heat | Liquid loops into each rack, because racks at about 120 kW are too dense for air [W] |
| Link | Typical speed | Reach |
|---|---|---|
| Chip to its own memory (HBM) | About 7 TB/s for one Ironwood chip [W] | Millimetres |
| GPU to GPU inside a rack | 900 GB/s each way for a Blackwell GPU [W] | Within one rack |
| Network card between racks | 800 Gbps, which is 100 GB/s [W] | Across the cluster |
| Datacentre network to a TPU | 100 Gbps per chip [W] | Across the site |
The link inside a rack is about nine times faster, per direction, than one 800 Gbps network card. [Calc] That gap is why the most talkative work, splitting a single layer across GPUs, stays inside one rack, and only the lighter exchanges cross the cluster network.