Transformer Processing from Query to Reply — Full Technical Summary

[smfadmin (OP)]

Transformer Processing — Complete Technical Summary

The pipeline:

Input Text
    ↓
Tokenization
    ↓
Embedding Layer
    ↓
──────────────────────────────────────────────
 Transformer Stack (80 layers)
──────────────────────────────────────────────
    ↓
Self‑Attention → Keys / Queries / Values
    ↓
Each token produces three vectors at every layer:
• Query Vector (Q): asks “What information should I pay attention to?”
• Key Vector (K): answers “Here is what I contain.”
• Value Vector (V): provides “Here is the actual information to mix in.”
Together, the Key and Value vectors are referred to as KV.
Attention scores come from comparing Queries to Keys, and those scores are used to weight the Values.
    ↓
Mixed Hidden States (context‑rich)
    ↓
Final Hidden State of Last Input Token
    ↓
W_out × h_last  →  Logits
    ↓
Softmax → Probability Distribution
    ↓
Sample Next Token
    ↓
Append to KV (Key Value) Cache
    ↓
Loop back to the "Final Hidden State of Last Input Token" for each output token until the EOS (End Of Sequence) token is generated OR max output length is reached.

1. Overview
Transformers process the user’s **query** and generate a **reply** using two different modes:

- Query: fixed sequence → processed in parallel 
- Reply: variable sequence → generated sequentially (autoregressive)

This asymmetry is fundamental to how modern language models operate.

---

2. Tokenization & Vocabulary
Text is split into **tokens**, not words. Tokens are sub‑word units chosen by a statistical tokenizer.

Examples: 
- “the” → 1 token 
- “transformers” → 2 tokens 
- “unbelievable” → 3 tokens 
- “🦘” → 1 token 
- “neurotransmitter” → “neuro” + “trans” + “mitter”

Vocabulary: 
The vocabulary is the complete list of tokens the model can output. 
Typical vocab sizes:

- GPT‑style: ~50,000 tokens 
- LLaMA‑style: ~32,000 tokens 
- Multilingual models: 65,000–250,000 tokens 

Each token has:

- a unique ID 
- an embedding vector 
- a row in W_out 
- a learned meaning

What the Vocabulary Contains
The model’s vocabulary is essentially is a set of symbols that it can read and produce; a complete internal database of all possible tokens it can output.
 
Every stage of the pipeline depends on it.

The vocabulary is the model's interface to language. It defines every symbol the model can read and every symbol it can produce. Transformer weights are the learned machinery that gives those symbols meaning in context.

It does not store whole words like a human dictionary — instead it stores the atomic text fragments the tokenizer has learned.

It is a fixed component of the model to be share and the parameters associated with that token have already been learned.

The knowledge is not in the vocabulary. The knowledge emerges from the billions of learned weights in the transformer.

It is not:

- a dictionary of definitions,
- a knowledge base,
- a database of facts,
- or the source of the model's intelligence.

Symbol: A symbol is an individual entry in the model's vocabulary representing a unit of text that the model can recognise and generate. A symbol may be a whole word (the, Australia), part of a word (trans, ing), punctuation (., ?), whitespace (Ġthe meaning "space + the"), an emoji (😊), or a special token such as BOS (Beginning Of Sequence) or EOS (End Of Sequence). Each symbol has a unique token ID, a learned embedding vector, and a corresponding row in W_out.

It contains:
- Whole‑word tokens for very common words (“the”, “and”, “Australia”). 
- Sub‑word fragments (“trans”, “form”, “ing”) used to build larger words. 
- Punctuation tokens (“.”, “,”, “!”, “?”). 
- Whitespace tokens such as “Ġthe” meaning “space + the”. 
- Emoji tokens (“😊”, “🧠”, “🦘”) which compress meaning into a single token. 
- Unicode/byte‑level tokens for rare characters, symbols, and multilingual text. 
- Special tokens (BOS, EOS, system markers).

Each token has:
- a unique ID 
- an embedding vector 
- a row in the output projection matrix (W_out) -- see point 6 further below, for an explanation of W_out
- a learned semantic relationship to other tokens

In short: the vocabulary is the model’s complete token database — the full set of building blocks it uses to read and generate text.

Important: Vocabulary contains **fragments**, not words.

---

3. Why Models Sometimes Output Emojis
Emojis are part of the vocabulary. 
They exist because:

- they appear frequently in training data 
- they compress meaning into a single token 
- they reduce token count compared to long phrases 
- they express tone, emotion, or emphasis efficiently 
- they improve alignment with human communication patterns

For example:

- “happy” → 1–2 tokens 
- “😊” → 1 token

Thus emojis are sometimes chosen because they are **high‑information, low‑token‑cost symbols**.

---

4. Query Processing (Parallel + Sequential)
Assume:

- 80 transformer layers 
- 1000‑token input sequence 
- 4096‑dimensional vectors 
- fp16 precision (2 bytes per float)

Parallel across tokens: 
All 1000 tokens are processed simultaneously inside each layer. 
This is possible because the input is a fixed, known sequence.

Sequential across layers: 
Layer 1 → Layer 2 → … → Layer 80 
Each layer waits for the previous one.

Hidden state size: 
4096 floats × 2 bytes = 8192 bytes per hidden state (~8 KB)

Key/value vector size: 
Same dimension → also ~8 KB each

Per token vector count: 
- 1 embedding 
- 80 hidden states 
- 80 key vectors 
- 80 value vectors 
Total = 241 vectors per token

Per token memory footprint: 
241 vectors × 8 KB ≈ ~1.9 MB per token

Total query memory (1000 tokens): 
1000 × 1.9 MB ≈ ~1.9 GB

This is why long‑context models require enormous GPU memory.

---

5. Hidden States
A **hidden state** is a high‑dimensional vector representing the model's internal understanding of a token at a given layer.

Each token has:

- 1 embedding (layer 0) 
- 1 hidden state per layer 
- 1 key per layer 
- 1 value per layer 

The final hidden state of the last input token is used to generate the first output token.

Self‑attention mixes information — but only in one direction.
(Self‑attention lets each token look at itself and every token before it in the sequence — never the tokens that come after. This is enforced by a causal mask, which blocks attention to future positions.
It computes attention scores that determine which earlier tokens contribute strongly to the current token's hidden state.
These scores are used to form weighted combinations of key and value vectors across all earlier tokens (including itself).
This allows the model to mix information from everything said so far, linking related concepts regardless of distance — while still respecting the left‑to‑right structure of language generation.)

Because of this masking:

- every token's hidden state contains information only from itself and everything before it

- the last token in the sequence is the only one that has seen the entire prompt

- its hidden state is therefore the most context‑rich, and serves as the "launch vector" for generating the first output token

The model computes:
logits = W_out × h_last -- see below for an explanation of W_out

This produces the probability distribution for the first output token or in other words, a list of probabilities over all possible next tokens the model could output.

5b. Worked Example — Self‑Attention and No Retroactive Rewriting
Query or prompt: "The bridge is safe. Actually, ignore that, it's condemned."

This is self‑attention in action — each token attending to other tokens within the same sequence (not a separate one, as in cross‑attention). Specifically:

- The hidden state at "safe" is computed and frozen the moment it's processed. At that point, self‑attention only lets it look at "The" and "bridge" — nothing after it exists yet as far as the mask is concerned. It has no idea a correction is coming, and it never finds out. 

- The hidden state at "condemned" uses self‑attention to look backward across the whole sequence so far — "bridge," "safe," "Actually," "ignore," "that" — computing attention scores against each of their key vectors and using those scores to pull in their value vectors. This is how it builds a representation that reflects the contradiction. 

- Nothing about "safe" is ever edited by this process. Self‑attention only ever writes into the token doing the attending — the correction isn't made to an earlier token's hidden state, it's made in a later one, which carries the fuller picture forward.

By the time the model reaches the last token in the prompt, self‑attention has let it accumulate the whole exchange — safe, then contradicted, then condemned — layer by layer. That final hidden state is what launches generation of the reply. The model reaches the right answer not by rewriting history, but because self‑attention lets the last position see everything before it, while everything before it stays fixed.

---

6. W_out (Output Projection Matrix)
W_out is the model’s final output projection matrix. 
It converts the hidden state into **logits**.

Shape:

vocab_size × model_dimension

Example:

- vocab size = 50,000 
- model dimension = 4096 

Then:

W_out = 50,000 × 4096

Each row corresponds to one token in the vocabulary.

When generating a token:

logits = W_out × h_current

This produces one logit per vocabulary entry (e.g., 50,000 logits).

---

7. Logits (Raw Scores Before Probability)
A **logit** is the model’s raw, unnormalized score for each possible next token.

- Not a probability 
- Can be any real number 
- Higher logit = model prefers that token more

Logits come from:

logits = W_out × h_last

This produces one logit per vocabulary entry (e.g., 50,000 logits).

Logits are fed into softmax to produce probabilities.

----

8. Softmax (Turning Logits Into Probabilities)
Softmax converts logits into a probability distribution:

softmax(z_i) = e^{z_i} / Σ e^{z_j}

This ensures:

- all probabilities are positive 
- they sum to 1 
- the highest logit becomes the highest probability 

The model then samples (or selects) the next token.

---

9. Sequential Reply Generation
Replies cannot be generated in parallel because each output token depends on all previous output tokens:

P(t_n | t_1 … t_{n-1})

Thus the model must generate:

1 → 2 → 3 → … → N

Each new token:

- produces its own hidden state 
- produces its own key/value vectors 
- appends them to the KV cache 
- becomes part of the context for the next token

---

10. KV Cache Anchoring
The KV cache built during the query phase is the **anchor** that keeps the reply aligned with the prompt.

During reply generation, each new token attends to:

- all query keys/values 
- all previously generated output keys/values 

This ensures:

- contextual grounding 
- relevance 
- coherence 
- memory of the prompt

The model does **not** reprocess the query. 
It only uses the KV cache.

---

11. Final Summary
- Query: fixed string → parallel token processing → sequential layer stack 
- Reply: variable string → strictly sequential autoregressive generation 
- Hidden states: many per token, final one used to start generation 
- W_out: converts hidden states into logits 
- Logits: raw scores for each possible next token 
- Softmax: converts logits into probabilities 
- Vocabulary: list of all tokens the model can output 
- Emojis: efficient, high‑information tokens 
- KV cache: anchors reply to query without reprocessing 
- Byte counts: hidden states ~8 KB each; full query often ~GB scale 
- No reverse pass, no cross‑checking, no parallel reply generation