Transformer Processing from Query to Reply — Full Technical Summary

[smfadmin (OP)]

Transformer Processing — Complete Technical Summary

0. Before Transformers — RNNs & LSTMs
Before self‑attention, sequence models were built on RNNs (Recurrent Neural Networks) and their improved variant, LSTMs (Long Short‑Term Memory networks).

The core idea: read one token at a time (as a human would do (mind the bottleneck, lol)), and carry a single running hidden state forward as a compressed summary of everything seen so far — token 1 → update state → token 2 → update state → token 3... — all the way through both the prompt and the reply. There was no separate parallel phase; everything was sequential, start to finish.

This had two major problems:

- Vanishing information: Since everything gets squeezed into one fixed‑size hidden state, information from early tokens gets overwritten and fades as the sequence gets longer — long‑range context was hard to retain.
- No parallelism: Token 2 couldn't be processed until token 1 was finished, so training and inference couldn't exploit GPUs' strength at doing thousands of operations simultaneously. Long sequences were slow to train.

The 2017 "Attention Is All You Need" paper by a team of researchers at Google solved both problems by dropping recurrence entirely: self‑attention lets every token look directly at every other token's key/value pair (no fading, no bottleneck), and — for a fixed input sequence — every token can be processed in parallel rather than one at a time. This is the origin of the parallel‑prefill / sequential‑reply split described below.

The pipeline:

      Input Text
           │
           ▼
      Tokenization
           │
           ▼
    Embedding Layer
           │
           ▼
=======================================================
                     Transformer Stack (80 Layers)
=======================================================
           │
           ▼
┌──────────────────────────────────────────────────────────────┐
│                           SELF-ATTENTION                             
│                                                                     
│  Each token produces three vectors at every layer:                   
│  • Query Vector (Q): Asks “What information should I pay           
│                      attention to?”                                 
│  • Key Vector (K):   Answers “Here is what I contain.”               
│  • Value Vector (V): Provides “Here is the actual information       
│                      to mix in.”                                     
│                                                                     
│  * Key and Value vectors are together referred to as KV.           
│  * Attention scores come from comparing Queries to Keys, and those   
│    scores are used to weight the Values.                             
└──────────────────────────────────────────────────────────────┘
           │
           ▼
  Mixed Hidden States (Context-Rich)
           │
           ▼
=======================================================
      PHASE 1: Query Processing (Parallel Prefill)
=======================================================
  Processes ALL prompt tokens simultaneously. Populates the KV Cache.
  Isolates the Final Hidden State of the LAST Input Token (h_last).
           │
           ▼
 ┌──► 🚀 CALCULATE LOGITS: W_out × h_current
 │                    │
 │                    ▼
 │                 Softmax ──► Probability Distribution
 │                    │
 │                    ▼
 │               Sample Next Token
 │                    │
 │                    ▼
 │         Append New K & V to KV Cache
 │                    │
 │                    ▼
 │    [Is token EOS (End Of SEquence) or Max Length reached?]
 │     ├── YES ──► 🛑 Terminate Generation
 │     └── NO  ──► 🔄 Keep Going (Proceed to Phase 2 below)
 │                    │
 │                    ▼
 │=====================================================
 │    PHASE 2: Reply Building (Sequential Decoding Loop)
 │=====================================================
 │  Takes ONLY the single newest token from the step above.
 │  Passes it through the layers, reading backward against the static KV Cache.
 │  Outputs a brand-new current hidden state (h_current).
 │                                        ▼
 └────────────────────┘
1. Overview & Architectural Asymmetry
Transformers process a user’s query and generate a reply using two fundamentally different modes of computation:

Phase 1: Query Processing (Prefill): The input string is fixed and known. All input tokens are processed simultaneously in parallel through the entire layer stack.

Phase 2: Reply Building (Decoding): The output string is variable and unknown. Tokens must be generated sequentially, one by one (autoregressively), via an internal feedback loop.

This asymmetry is the foundational driving force behind modern LLM engineering and hardware demands.

2. Tokenization & Vocabulary Database
Text is split into **tokens**, not words. Tokens are sub‑word units chosen by a statistical tokenizer.

Examples: 
- “the” → 1 token 
- “transformers” → 2 tokens 
- “unbelievable” → 3 tokens 
- “🦘” → 1 token 
- “neurotransmitter” → “neuro” + “trans” + “mitter”

Vocabulary: 
The vocabulary is the complete list of tokens the model can output. 
Typical vocab sizes:

- GPT‑style: ~50,000 tokens 
- LLaMA‑style: ~32,000 tokens 
- Multilingual models: 65,000–250,000 tokens 

Each token has:

- a unique ID 
- an embedding vector 
- a row in W_out 
- a learned meaning

What the Vocabulary Contains
The model’s vocabulary is essentially a set of symbols that it can read and produce; a complete internal database of all possible tokens it can output.
 
Every stage of the pipeline depends on it.

The vocabulary is the model's interface to language. It defines every symbol the model can read and every symbol it can produce. Transformer weights are the learned machinery that gives those symbols meaning in context.

It does not store whole words like a human dictionary — instead it stores the atomic text fragments the tokenizer has learned.

It is a fixed component of the model to be shared and the parameters associated with that token have already been learned.

The knowledge is not in the vocabulary. The knowledge emerges from the billions of learned weights in the transformer.

It is not:

- a dictionary of definitions,
- a knowledge base,
- a database of facts,
- or the source of the model's intelligence.

Symbol: A symbol is an individual entry in the model's vocabulary representing a unit of text that the model can recognise and generate. A symbol may be a whole word (the, Australia), part of a word (trans, ing), punctuation (., ?), whitespace (Ġthe meaning "space + the"), an emoji (😊), or a special token such as BOS (Beginning Of Sequence) or EOS (End Of Sequence). Each symbol has a unique token ID, a learned embedding vector, and a corresponding row in W_out.

It contains:
- Whole‑word tokens for very common words (“the”, “and”, “Australia”). 
- Sub‑word fragments (“trans”, “form”, “ing”) used to build larger words. 
- Punctuation tokens (“.”, “,”, “!”, “?”). 
- Whitespace tokens such as “Ġthe” meaning “space + the”. 
- Emoji tokens (“😊”, “🧠”, “🦘”) which compress meaning into a single token. 
- Unicode/byte‑level tokens for rare characters, symbols, and multilingual text. 
- Special tokens (BOS, EOS, system markers).

Each token has:
- a unique ID 
- an embedding vector 
- a row in the output projection matrix (W_out) -- see point 6 further below, for an explanation of W_out
- a learned semantic relationship to other tokens

In short: the vocabulary is the model’s complete token database — the full set of building blocks it uses to read and generate text.

Important: Vocabulary contains **fragments**, not words.

3. Why Models Sometimes Output Emojis
Emojis are part of the vocabulary. 
They exist because:

- they appear frequently in training data 
- they compress meaning into a single token 
- they reduce token count compared to long phrases 
- they express tone, emotion, or emphasis efficiently 
- they improve alignment with human communication patterns

For example:

- “happy” → 1–2 tokens 
- “😊” → 1 token

Thus emojis are sometimes chosen because they are **high‑information, low‑token‑cost symbols**.

4. Query Processing (Parallel + Sequential)
Assume:
- 80 transformer layers 
- 1000‑token input sequence 
- 4096‑dimensional vectors 
- fp16 precision (2 bytes per float)

Parallel across tokens: 
All 1000 tokens are processed simultaneously inside each layer. 
This is possible because the input is a fixed, known sequence.

Sequential across layers: 
Layer 1 → Layer 2 → … → Layer 80 
Each layer waits for the previous one.

Hidden state size: 
4096 floats × 2 bytes = 8192 bytes per hidden state (~8 KB)

Key/value vector size: 
Same dimension → also ~8 KB each

Two different memory numbers — don't conflate them:

① Transient activation memory (used, then discarded): 
Each token produces 1 embedding and 80 hidden states as it moves through the stack — but these are only needed momentarily, to compute the next layer's output. Once layer N+1 has consumed layer N's hidden state, layer N's is discarded. It is not kept around.

② Persistent KV cache (kept for the entire generation): 
Only the keys and values are retained — because reply generation later needs to attend back to them. Nothing else survives.

Per token, retained (KV cache only): 
- 80 key vectors 
- 80 value vectors 
Total = 160 vectors per token

Per token KV cache footprint: 
160 vectors × 8 KB ≈ ~1.28 MB per token

Total KV cache (1000 tokens): 
1000 × 1.28 MB ≈ ~1.28 GB

This is the number that actually sits in GPU memory for the duration of generation — embeddings and hidden states come and go layer by layer, but the KV cache persists until the reply is finished. It's also why long‑context models demand enormous GPU memory: the cache grows linearly with sequence length, for every token, for as long as generation continues.

Note: Many production models today use grouped‑query or multi‑query attention, where several query heads share a single K/V head. This shrinks the cache well below the naive per‑head math shown here — often by 4–8× — which is one of the main reasons real‑world deployments can serve long contexts at all.

5. Hidden States
A **hidden state** is a high‑dimensional vector representing the model's internal understanding of a token at a given layer.

Each token has:

- 1 embedding (layer 0) 
- 1 hidden state per layer 
- 1 key per layer 
- 1 value per layer 

The final hidden state of the last input token is used to generate the first output token.

Self‑attention mixes information — but only in one direction.
(Self‑attention lets each token look at itself and every token before it in the sequence — never the tokens that come after. This is enforced by a causal mask, which blocks attention to future positions.
It computes attention scores that determine which earlier tokens contribute strongly to the current token's hidden state.
These scores are used to form weighted combinations of key and value vectors across all earlier tokens (including itself).
This allows the model to mix information from everything said so far, linking related concepts regardless of distance — while still respecting the left‑to‑right structure of language generation.)

Because of this masking:

- every token's hidden state contains information only from itself and everything before it

- the last token in the sequence is the only one that has seen the entire prompt

- its hidden state is therefore the most context‑rich, and serves as the "launch vector" for generating the first output token

The model computes:
logits = W_out × h_last -- see below for an explanation of W_out

This produces the probability distribution for the first output token or in other words, a list of probabilities over all possible next tokens the model could output.

5b. Worked Example — Self‑Attention and No Retroactive Rewriting
Query or prompt: "The bridge is safe. Actually, ignore that, it's condemned."

This is self‑attention in action — each token attending to other tokens within the same sequence (not a separate one, as in cross‑attention). Specifically:

- The hidden state at "safe" is computed and frozen the moment it's processed. At that point, self‑attention only lets it look at "The" and "bridge" — nothing after it exists yet as far as the mask is concerned. It has no idea a correction is coming, and it never finds out. 

- The hidden state at "condemned" uses self‑attention to look backward across the whole sequence so far — "bridge," "safe," "Actually," "ignore," "that" — computing attention scores against each of their key vectors and using those scores to pull in their value vectors. This is how it builds a representation that reflects the contradiction. 

- Nothing about "safe" is ever edited by this process. Self‑attention only ever writes into the token doing the attending — the correction isn't made to an earlier token's hidden state, it's made in a later one, which carries the fuller picture forward.

By the time the model reaches the last token in the prompt, self‑attention has let it accumulate the whole exchange — safe, then contradicted, then condemned — layer by layer. That final hidden state is what launches generation of the reply. The model reaches the right answer not by rewriting history, but because self‑attention lets the last position see everything before it, while everything before it stays fixed.

6. W_out (Output Projection Matrix)
W_out is the model’s final output projection matrix. 
It converts the hidden state into **logits**.

Every vocabulary symbol has it's own logit and the logits are recalculated at every step of the reply process. Softmax transforms into the logit into a probability value in a transient array called the probability distribution vector — a plain array of 50,000 numbers (one per vocab entry), sitting in GPU memory, produced fresh each step, so the array gets rebuilt for the sole purpose of influencing the generation of the next token correctly.

Shape:

vocab_size × model_dimension

Example:

- vocab size = 50,000 
- model dimension = 4096 

Then:

W_out = 50,000 × 4096

Each row corresponds to one token in the vocabulary.

When generating a token:

logits = W_out × h_current

This produces one logit per vocabulary entry (e.g., 50,000 logits).

7. Logits (Raw Scores Before Probability)
A **logit** is the model’s raw, unnormalized score for each possible next token; it is fixed, specific, meaningful — just not yet forced into probability-shape.

- Not a probability 
- Can be any real number 
- Higher logit = model prefers that token more

Logits come from:

logits = W_out × h_last

This produces one logit per vocabulary entry (e.g., 50,000 logits).

Logits are fed into softmax to produce probabilities.

8. Softmax (Turning Logits Into Probabilities)
Softmax converts logits into a probability distribution:

softmax(z_i) = e^{z_i} / Σ e^{z_j}

This ensures:

- all probabilities are positive 
- they sum to 1 
- the highest logit becomes the highest probability 

The model then samples (or selects) the next token.

=======================================================

9. Sequential Reply Generation
Replies cannot be generated in parallel because each output token depends on all previous output tokens:

P(t_n | t_1 … t_{n-1})

Thus the model must generate:

1 → 2 → 3 → … → N

Each new token:

- produces its own hidden state 
- produces its own key/value vectors 
- appends them to the KV cache 
- becomes part of the context for the next token

10. KV Cache Anchoring
The KV cache built during the query phase is the **anchor** that keeps the reply aligned with the prompt.

During reply generation, each new token attends to:

- all query keys/values 
- all previously generated output keys/values 

This ensures:

- contextual grounding 
- relevance 
- coherence 
- memory of the prompt

The model does **not** reprocess the query. 
It only uses the KV cache.

11. Final Summary
- Query: fixed string → parallel token processing → sequential layer stack 
- Reply: variable string → strictly sequential autoregressive generation 
- Hidden states: many per token, final one used to start generation 
- W_out: converts hidden states into logits 
- Logits: raw scores for each possible next token 
- Softmax: converts logits into probabilities 
- Vocabulary: list of all tokens the model can output 
- Emojis: efficient, high‑information tokens 
- KV cache: anchors reply to query without reprocessing 
- Byte counts: hidden states ~8 KB each; full query often ~GB scale 
- No reverse pass, no cross‑checking, no parallel reply generation