Scaling Laws

[Chip (OP)]

Scaling Laws

Scaling laws describe how model performance improves predictably as you increase:

• Model size (parameters)
• Training data
• Compute (GPU time)

This is one of the key reasons modern AI works at all.

Instead of being chaotic or unpredictable, performance follows smooth mathematical curves.

---

1. Why bigger models suddenly worked

Early neural networks were small and unstable. 
They improved slowly and often hit performance ceilings.

Then researchers discovered:

Code: [Select]

Scaling up model size + data + compute → consistent performance gains

No architectural breakthrough was required at first — just scale.

Key insight:

• Small models underfit reality
• Large models start capturing structure in data
• Very large models generalise surprisingly well

This led to a phase shift in capability.

---

2. Parameter scaling

Parameters are the internal weights of a model.

Scaling parameters means:

Code: [Select]

More weights → more representational capacity

Effects:

• Better pattern recognition
• More nuanced representations
• Improved generalisation (up to a point)

Empirical finding:

Code: [Select]

Loss decreases predictably as parameters increase

This relationship is smooth and surprisingly stable across architectures.

---

3. Data scaling

More parameters require more data.

Otherwise:

Code: [Select]

Large model + small data = overfitting

So scaling laws include dataset growth:

• More diverse text
• More languages
• More domains (code, science, dialogue)

Key principle:

Code: [Select]

Data and model size must grow together

Otherwise gains plateau.

---

4. Compute scaling

Compute is the practical limit: GPU time and energy.

Training cost scales roughly with:

Code: [Select]

Parameters × Data × Training steps

So scaling models is expensive:

• Requires massive GPU clusters
• Weeks or months of training
• Huge energy costs

But compute directly determines how far scaling can go.

---

5. Emergent behaviour

As models scale, new abilities appear that were not explicitly programmed.

Examples:

• In-context learning
• Better reasoning
• Code generation
• Translation abilities
• Instruction following

Key idea:

Code: [Select]

Capabilities do not increase linearly with size
They often appear suddenly at thresholds

This is called emergence.

It is not magic — it is the result of crossing complexity thresholds in representation space.

---

6. Why GPT-3 changed everything

GPT-3 was a turning point because it demonstrated:

• Massive scale works reliably
• Few-shot learning emerges naturally
• General-purpose language ability becomes strong

Before GPT-3:

• Models were task-specific
• Fine-tuning was required for most tasks

After GPT-3:

Code: [Select]

One model → many tasks via prompting

This shifted AI from:

Code: [Select]

Task-specific systems
→ general-purpose foundation models

---

7. Scaling law intuition

Scaling laws can be thought of as:

Code: [Select]

More scale → smoother approximation of the underlying data distribution

As scale increases:

• Noise reduces
• Patterns sharpen
• Rare structures become learnable

The model becomes a better statistical compressor of reality.

---

Key Insight

The surprising discovery is not just that bigger is better.

It is that:

Code: [Select]

Performance improves in a predictable, mathematical way with scale

This predictability allowed AI to become an engineering discipline rather than experimental guesswork.

And it explains why modern progress looked sudden:

• Once scaling laws were found
• You could reliably build better models just by scaling resources