How Many Copies of a Model Run, How They Stay in Sync, and Why You Can't Just Keep Training It

This document answers three connected questions about a large AI model running in a datacentre:

1. Is a model one big program, or many copies?
2. How do many copies train on different data without conflicting?
3. Why not simply switch a live model into "training mode" and feed it new knowledge?

All of it comes from my own background knowledge, not from the transcript or a web search.

1. One model, many running copies

A trained model is one set of numbers, called its weights. They are saved as a single checkpoint, though for a large model that checkpoint is split across many files. What runs in the datacentre is copies of those weights loaded into the memory of GPUs. A running copy is called a replica or an instance.

How the copies are used differs completely between training and inference.

2. Training: many copies that behave as one

Training runs one model, but not on one machine.

• Replicas: several identical copies of the model train at the same time, each on a different batch of data. This is called data parallelism.
• Sharding: each replica is too big for one GPU, so it is cut up. Different layers, or parts of a layer, sit on different GPUs. A single replica can fill a whole rack.
• Result: when training ends, one set of weights is saved.

There is no master copy

Nobody holds a central set of weights that the others report to. Every replica holds its own full copy, and the copies stay identical by design:

1. Same start: every replica begins with exactly the same weights.
2. Different data: each replica processes its own batch and works out its own gradient, the adjustment it would make to the weights.
3. Average: an all-reduce combines the gradients into one average. This is the transcript's 0.7, 0.9, 0.6, 0.8 → 0.75 example, repeated for billions of numbers.
4. Same update: every replica receives the same averaged gradient and applies the same update with the same formula. Starting identical and receiving identical changes, they end identical.

Why there are no conflicts

Conflicts happen when several writers change the same data independently. Here nobody writes independently. Each replica only produces a suggestion, and the suggestions are averaged before anyone changes anything. The averaged gradient is also what one very large batch of data would have produced, so N replicas on different data behave like one model training on N times as much data per step.

What does go wrong

Problem	What happens	How it is handled
Waiting for the slowest	A step doesn't finish until every replica reports, so one slow GPU or jammed link delays everyone	This is why the network matters so much (see the earlier documents)
Failures	Across thousands of chips, something breaks regularly	The job restarts from the last saved checkpoint, so checkpoints are written often
Tiny drift	Computer arithmetic isn't perfectly repeatable, so copies can differ in the last decimal places	Usually harmless, with checks to catch anything larger
Memory limits	A full copy plus its training state may not fit on one GPU	FSDP (Fully Sharded Data Parallel), related to ZeRO (Zero Redundancy Optimizer), keeps only a slice of the weights on each GPU and fetches the rest when needed. It still has no master, only more network traffic

The older alternative

Older designs used a parameter server, a central machine holding the weights, with workers sending it updates without waiting for each other. That does create conflicts, because updates arrive based on weights that have already moved on. Large language model training mostly avoids it in favour of the synchronous averaging above.

3. Inference: many independent copies serving users

Once the model is finished, the same weights run as many separate instances.

• Replicas behind a load balancer: each replica is a full copy of the model on its own group of GPUs, often one rack or part of one. Serving more users means starting more replicas. Each replica handles many users at once by batching their requests together.
• A mix of models: a service usually runs several models side by side. A large model handles hard questions, smaller and cheaper ones handle simple work, and older versions stay in use. Each has its own pool of replicas, and older GPUs often serve the smaller ones.
• Packaging: a replica is typically a software container started on a group of GPUs. It must load the weights from storage into GPU memory before it can serve anyone, which takes minutes for a very large model.

Two special cases

• Mixture-of-experts models are one model made of many specialist sub-networks spread across GPUs. For each word only a few experts activate. This creates the all-to-all traffic described in the protocols document.
• Prefill and decode pools: the same weights can be loaded twice, once on GPUs tuned for reading prompts (prefill) and once on GPUs tuned for writing replies (decode). The conversation's working memory, the KV cache, is passed between the two.

4. Why you can't just keep training the live model

Continuing to train a model on new data is possible and routine. It is called continued pre-training or fine-tuning. The problems are practical.

There is no "training mode" switch

The replicas serving users hold frozen weights. Training also needs the gradients and the optimiser's bookkeeping alongside the weights. A common rule of thumb is about 16 bytes per parameter for training against about 2 for serving, roughly eight times the memory. So training runs on a separate cluster, and the result is a new version of the model, not a live edit of the old one. Every serving replica then has to be reloaded with the new weights.

What goes wrong in the training itself

• Forgetting: pushing new data through a trained model changes weights that old knowledge depends on. Skills in unrelated areas can quietly degrade, which is called catastrophic forgetting. The usual defences are a low learning rate and mixing old data back in, and both slow the learning of anything new.
• Little to gain in settled fields: if a discipline is well established, the model has already seen it thousands of times. More data of the same kind adds little, and too many passes over the same material makes the model memorise it instead of understanding it.
• No sense of which source is right: the weights keep no record of where a fact came from. If new data contradicts old data, the model blends them and can't say which is authoritative.
• Skewed by repetition: duplicated or low-quality text counts as extra votes for whatever it says.

What goes wrong around it

• Regressions: a model that was carefully tested and tuned to behave safely can lose that tuning after more training, so the full set of evaluations has to be re-run.
• Cost: a retraining run occupies a large part of a cluster for days or weeks.

The cheaper route for new information

If the goal is for the model to know a new paper or a changed rule, retrieval is usually better. The system looks up the relevant documents when a question is asked and puts them in front of the model, so the weights never change. Weight training suits changing how the model behaves or writes, or teaching it a whole domain it knows poorly.

Glossary

Term	Plain meaning
Weights	The billions of numbers that make up a trained model
Checkpoint	A saved snapshot of the weights, used for recovery or release
Replica / instance	One running copy of the model
Gradient	The adjustment a replica would make to the weights after seeing its batch of data
All-reduce	The step that averages every replica's gradient and returns the result to all of them
Data parallelism	Many replicas training on different data at once
Sharding	Cutting one copy of the model across several GPUs
Parameter server	An older design with a central machine holding the weights
Fine-tuning	Further training of a finished model on new data
Catastrophic forgetting	Loss of old abilities after training on new data
Retrieval	Looking up documents at question time instead of changing the weights
KV cache	The model's working memory of the conversation so far