LLM 101 · Section 2. The transformer: why it replaced RNNs, how attention and self-attention work, multi-head attention, the three model families, position and context length, and the KV cache.
Before transformers (2017), the standard model for text was the RNN (recurrent neural network). It reads one token at a time and carries a fixed-size memory, the hidden state, from each step to the next. A transformer processes all positions of the text at once.
RNN: must go in order
tok 1 → tok 2 → tok 3 → … → tok 1000
1,000 steps, each waiting for the one before
Transformer: all positions at once
tok 1 tok 2 tok 3 … tok 1000
↓ ↓ ↓ ↓
one step, every token processed together
For a fact at token 1 to matter at token 1,000, it has to survive 999 rewrites of one fixed-size memory, and mostly it doesn't. Training makes it worse. The gradient, the signal that tells the model which numbers to adjust, is multiplied by one factor per step on its way back. Slightly below 1 and it vanishes; slightly above 1 and it explodes:
print(f"0.9 multiplied 100 times: {0.9 ** 100:.5f}")
print(f"1.1 multiplied 100 times: {1.1 ** 100:,.0f}")
0.9 multiplied 100 times: 0.00003 1.1 multiplied 100 times: 13,781
These are the vanishing and exploding gradient problems. LSTMs, a gated kind of RNN, fixed much of the decay.
Step 500 needs step 499's output, so the steps can't run at the same time. GPUs are fast only at work that can run in parallel, so training an RNN on a large corpus would have taken years of wall-clock time. LSTMs didn't fix this: they still read one token at a time.
This second problem is the one that decided it. What the transformer really removed was the sequential dependency during training, which made it possible to train on thousands of GPUs at once.
Transformers are parallel during training, because the whole text is already known and every position can be scored at once. Generation is still one token per pass (§1). Most of the later sections follow from this split.
exercises/llm-101/02/01_sequential_vs_parallel.py — times the same matrix work done one row after another (like an RNN) and all at once.
one after another : 13.53 ms all at once : 3.34 ms (4x faster)
Try: rewrite one_after_another so it doesn't need the loop. Result: you can't: each h needs the previous one. That dependency is what stopped RNNs using a GPU.
On a laptop CPU the gap is modest and noisy (about 4–5× here); on a GPU, which runs thousands of independent operations at once, it is far larger.
Attention lets every token look at every other token, decide how relevant each one is, and mix them into itself accordingly. The relevance numbers are called attention weights; for each token they add up to 1.
"The animal didn't cross the street because it was too tired." → it = the animal "The animal didn't cross the street because it was too wide." → it = the street
Changing one word at the end flips what it refers to. Whatever processes it has to reach back and pull in the right earlier word. An RNN can only hope the right word is still in its memory. Attention looks at every token directly.
With attention, token 900 reaching token 3 is one operation, the same as reaching token 899. In an RNN, that information had to survive 897 rewrites of a fixed-size memory.
Attention was introduced in 2014 for translation, added on top of RNNs. The RNN compressed the whole input sentence into one summary vector before producing the output, and past about 25–30 words that summary couldn't hold enough. Attention let each output word take a fresh weighted mix of all the input words: to write "लाइट" for "Turn off the lights", look mostly at lights. Transformers kept the attention and dropped the RNN.
exercises/llm-101/02/02_what_it_attends_to.py — prints what GPT-2 attends to from the word "it" in both sentences.
# Section 02: look inside a real model and see which words "it" pulls from.
# Predict first: will "it" attend differently in the two sentences?
# (They differ only in the last word: "tired" versus "wide".)
import torch
from transformers import AutoModel, AutoTokenizer
tok = AutoTokenizer.from_pretrained("gpt2", clean_up_tokenization_spaces=True)
model = AutoModel.from_pretrained("gpt2", output_attentions=True, attn_implementation="eager")
for text in ["The animal didn't cross the street because it was too tired",
"The animal didn't cross the street because it was too wide"]:
ids = tok(text, return_tensors="pt").input_ids
words = [tok.decode(i).strip() for i in ids[0]]
look_from = words.index("it")
with torch.no_grad():
attn = torch.stack(model(ids).attentions) # [layer, batch, head, from, to]
w = attn[:, 0, :, look_from, :].mean(dim=(0, 1)) # average over 12 layers x 12 heads
w[0] = 0 # drop the first token (see the gotcha)
w = w / w.sum()
print(text)
for i in torch.argsort(w, descending=True)[:4]:
print(f" {words[i]:<8} {w[i]:.2f}")
print()
# Then try: set look_from = len(words) - 1, so you measure from the last word instead.
This one needs torch, transformers, which can't run in a browser. Run it locally:
cd ~/Downloads/ai-engineer-notes/exercises source .venv/bin/activate python llm-101/02/02_what_it_attends_to.py
The animal didn't cross the street because it was too tired because 0.31 it 0.19 animal 0.10 cross 0.10 The animal didn't cross the street because it was too wide because 0.31 it 0.19 animal 0.10 cross 0.10
Try: set look_from = len(words) - 1 to measure from the last word instead. Result: now the sentences differ, but attention mostly goes to nearby words (too 0.20, tired 0.18, was 0.12).
The two sentences give identical weights from "it" (because 0.31, it 0.19, animal 0.10): GPT-2 is a decoder, so at "it" it hasn't seen "tired" or "wide" yet (section 5). Averaged attention weights are also not a readable "this refers to that" arrow; meaning is built up across all 12 layers.
Self-attention is attention within one sentence: each word is rebuilt as a weighted mix of all the words around it. It turns a static embedding (one vector per word, whatever the context) into a contextual one. It works in three steps: score, convert to weights, mix.
Each word is a list of numbers (its embedding). The numbers here are picked by hand to be readable; a trained model learns its own. Each word also has a fourth number, 0 for all three, which is left out.
water money generic the [ 0.3 0.3 2.5 ] filler word river [ 2.6 0.0 0.5 ] very water bank [ 1.6 1.6 0.8 ] equally both
bank starts as equally "water" and "money". Self-attention should push it towards "water" because of river.
A dot product multiplies two lists position by position and adds the results. Large numbers in the same positions give a large score.
bank·the = 1.6×0.3 + 1.6×0.3 + 0.8×2.5 = 2.96 bank·river = 1.6×2.6 + 1.6×0.0 + 0.8×0.5 = 4.56 ← shared "water" bank·bank = 1.6×1.6 + 1.6×1.6 + 0.8×0.8 = 5.76
Divide each score by √d, where d is the number of values per vector (√4 = 2 here), then apply softmax so the weights are positive and add up to 1:
score ÷ 2 e^x weight
the 2.96 1.48 4.39 0.137
river 4.56 2.28 9.78 0.306
bank 5.76 2.88 17.81 0.557
───── ─────
31.98 1.000
The new bank is the weighted average of the three words, one position at a time:
water = 0.137×0.3 + 0.306×2.6 + 0.557×1.6 = 1.73 money = 0.137×0.3 + 0.306×0.0 + 0.557×1.6 = 0.93
| water | money | ||
|---|---|---|---|
bank on its own | 1.60 | 1.60 | ambiguous |
| "the river bank" | 1.73 | 0.93 | leans water |
| "the savings bank" | 0.93 | 1.73 | leans money |
Same word, same three steps; only the neighbour changed (savings is river with its water and money values swapped). The whole calculation in NumPy:
import numpy as np
E = np.array([[0.3, 0.3, 2.5, 0.0], # the
[2.6, 0.0, 0.5, 0.0], # river
[1.6, 1.6, 0.8, 0.0]]) # bank
scores = E @ E[2] # step 1: bank against every word
w = np.exp(scores / np.sqrt(4))
w = w / w.sum() # step 2: softmax
new_bank = w @ E # step 3: weighted mix
print("weights:", np.round(w, 3))
print("bank before:", E[2][:2], " after:", np.round(new_bank[:2], 2))
weights: [0.137 0.306 0.557] bank before: [1.6 1.6] after: [1.73 0.93]
Softmax exaggerates differences between scores. Without the division, bank's weights become 0.045 / 0.221 / 0.734: less mixing, more copying itself. The bigger reason is size: the dot product of two random vectors of length d has a variance of about d.
import numpy as np
rng = np.random.default_rng(0)
for d in (4, 64, 512):
q, k = rng.standard_normal((2, 100_000, d))
s = (q * k).sum(axis=1)
print(f"d = {d:3}: variance {s.var():6.1f} after dividing by sqrt(d): {(s / np.sqrt(d)).var():.2f}")
d = 4: variance 4.0 after dividing by sqrt(d): 1.01 d = 64: variance 64.1 after dividing by sqrt(d): 1.00 d = 512: variance 514.2 after dividing by sqrt(d): 1.00
With 512 values, typical scores are about 23 away from zero, softmax gives nearly 1.0 and 0.0, and the small weights stop learning because their gradients vanish. Dividing by √d brings the variance back to about 1 for any size. Strictly, d is the length of the key vector (below).
In the example, each word used its own embedding for everything. That version has nothing to learn. A real model multiplies each embedding by three learned matrices, WQ, WK and WV, shared by all words:
| Vector | Role | Used in |
|---|---|---|
| query (Q) | what this word is looking for | step 1, as the word doing the scoring |
| key (K) | what this word offers to be matched on | step 1, as the word being scored |
| value (V) | what this word hands over if chosen | step 3, the mix |
The names come from a dictionary lookup: d[query] finds the matching key and returns its value. In matrix form the whole thing is softmax(Q·Kᵀ / √d_k)·V, which is why the paper calls it scaled dot-product attention.
One difference from the example: real transformers add the mixed result to the original embedding (a residual connection) instead of replacing it.
exercises/llm-101/02/03_self_attention.py — computes the "river bank" example step by step.
the score 2.96 share 0.137 river score 4.56 share 0.306 bank score 5.76 share 0.557 bank before: water 1.60 money 1.60 bank after : water 1.73 money 0.93
Try: replace river with savings = [0.0, 2.6, 0.5, 0.0]; then, separately, delete / np.sqrt(4). Result: with savings, water and money flip (0.93 / 1.73); without the division the weights become 0.045 / 0.221 / 0.734 and bank blends less.
One attention calculation produces one set of weights, so it can capture one kind of relationship. A token usually needs several at once:
"The keys to the cabinet were on the table." To get "were" right you need: which noun is the subject → "keys" (not "cabinet") what tense this is → the wider sentence what it connects to → "on the table"
Multi-head attention runs several attention calculations (heads) in parallel, each with its own WQ, WK and WV, so each can learn a different notion of relevance.
| Model | Values per token | Heads × size per head |
|---|---|---|
| Original transformer (2017) | 512 | 8 × 64 |
| A 4,096-wide model | 4,096 | 32 × 128 |
| GPT-3 | 12,288 | 96 × 128 |
Splitting the width this way costs about the same as one full-width head: 32 heads of 128 is the same total width as one head of 4,096. You get several kinds of relationship for roughly the price of one.
Researchers inspecting trained models find heads that track grammatical subjects, heads that match quotes to speakers, and heads that mostly look at the previous token. But this is a useful story, not a guarantee: heads are found by training, not assigned, many do things no one can name, and removing a supposedly critical head often changes little.
exercises/llm-101/02/04_two_heads.py — runs two heads over "the river bank" and shows they weight the words differently.
one big head the 0.14 river 0.31 bank 0.56 head A (meaning) the 0.03 river 0.33 bank 0.64 head B (generic) the 0.59 river 0.19 bank 0.22
Try: change the to [0.3, 2.5, 2.5, 0.0], giving it a large money value. Result: head A's weight for "the" jumps from 0.03 to 0.30; head B is unchanged because it never sees the money value.
The script gives each head a fixed slice of the numbers so the effect is easy to see. Real heads see the whole vector through learned projections.
The three transformer families differ in one thing: which tokens each token is allowed to look at. Everything else is the same.
| Family | Each token sees | Trained to | Good at | Examples |
|---|---|---|---|---|
| Encoder-only | everything, both directions | fill in hidden words | classifying, embeddings | BERT, e5 |
| Decoder-only | itself and earlier tokens only | predict the next token | generating text | GPT, Claude, Llama |
| Encoder-decoder | encoder: everything; decoder: left only | rebuild hidden spans | fixed input → output (translation, summaries) | T5, BART |
A decoder is enforced by a causal mask: a matrix of 0s and minus infinity added to the scores after dividing by √d and before softmax. Softmax turns minus infinity into exactly 0, so later tokens get zero weight and each row still adds up to 1. Setting weights to 0 after softmax would break that. The same three words as section 3, every row:
import numpy as np
E = np.array([[0.3, 0.3, 2.5, 0.0], [2.6, 0.0, 0.5, 0.0], [1.6, 1.6, 0.8, 0.0]])
scores = E @ E.T / np.sqrt(4)
def softmax_rows(x):
x = np.exp(x - x.max(axis=1, keepdims=True))
return x / x.sum(axis=1, keepdims=True)
mask = np.triu(np.full((3, 3), -np.inf), k=1) # minus infinity above the diagonal
print("encoder (sees everything):")
print(np.round(softmax_rows(scores), 2))
print("decoder (looks left only):")
print(np.round(softmax_rows(scores + mask), 2))
encoder (sees everything): [[0.78 0.09 0.14] [0.06 0.73 0.21] [0.14 0.31 0.56]] decoder (looks left only): [[1. 0. 0. ] [0.08 0.92 0. ] [0.14 0.31 0.56]]
In the decoder, the first word can only see itself (1.00, 0, 0). The last row is the same in both, because the last word already sees everything to its left. The mask only removes information from earlier words.
During training, a model predicting token 5 that can see token 5 is just copying the answer, which teaches nothing usable at generation time, when nothing to the right exists yet. CampusX calls this data leakage.
The decoder in an encoder-decoder model has one extra attention step, cross-attention: its queries come from the output being written, and its keys and values from the encoder's reading of the input. This is the translation setup from section 2.
The split mirrors §1: representation models are encoders, generative models are decoders.
exercises/llm-101/02/05_masks.py — prints the full attention weights for an encoder and a decoder.
encoder: sees everything the 0.78 0.09 0.14 river 0.06 0.73 0.21 bank 0.14 0.31 0.56 decoder: looks left only the 1.00 0.00 0.00 river 0.08 0.92 0.00 bank 0.14 0.31 0.56
Try: add a fourth word, "flooded" = [2.4, 0.0, 0.6, 0.0], and change the 3 in triu_indices to 4. Result: now only the flooded row matches between the two; bank's row changes because the encoder can see flooded and the decoder can't.
Almost every large general-purpose language model is decoder-only. Three reasons, in order of importance.
Underneath all of them: next-token prediction needs no labels. The text is its own answer key (every next token is the target), so any text on the internet can be training data without anyone annotating it.
From one 512-token passage, a decoder gets a prediction at every position, while an encoder-style model hides about 15% of tokens (BERT's rate) and predicts only those:
n = 512
decoder = n - 1
encoder = n * 0.15
print(f"decoder-only: {decoder} predictions")
print(f"encoder-style: {encoder:.0f} predictions")
print(f"ratio: {decoder / encoder:.1f}x")
decoder-only: 511 predictions encoder-style: 77 predictions ratio: 6.7x
High-quality text is limited, so getting 6.7× more learning from the same text is decisive. All 511 predictions happen in one pass because of the causal mask (section 5) and teacher forcing: during training each position's input is the true previous token from the data, not the model's own guess, so every input is known in advance. Without it, a 300-token target would need 301 passes one after another (CampusX's count).
Any task can be written as text completion: classify, translate, summarise. An encoder needs a task-specific output layer added; a decoder only needs a different prompt.
One stack, one objective, one mask: fewer parts to tune when a training run costs millions. Inference stays simple too: a decoder only ever appends tokens on the right, so earlier keys and values never change and can be cached (the KV cache, section 8).
For search and classification, a small encoder beats a large decoder at a fraction of the cost, which is the two-model setup from §1. Decoder-only dominates general-purpose models, not everything.
exercises/llm-101/02/06_training_signal.py — counts the training signals each design gets from a real passage, using a real tokenizer.
# Section 06: count how much a model learns from one passage.
# Predict first: how many predictions does each design get from this text?
import tiktoken
text = ("Attention lets every token look at every other token in the sentence, "
"decide how relevant each one is, and mix them into a new representation. "
"Stacking this many times, with ordinary layers in between, is what a "
"transformer is.")
n = len(tiktoken.get_encoding("cl100k_base").encode(text))
print(f"passage: {n} tokens")
print(f"decoder-only, predict every next token : {n - 1} training signals")
print(f"encoder-style, predict 15% hidden ones : {round(n * 0.15)} training signals")
print(f"ratio : {(n - 1) / (n * 0.15):.1f}x")
# Then try: paste a paragraph from these notes into `text` and recount.
This one needs tiktoken, which can't run in a browser. Run it locally:
cd ~/Downloads/ai-engineer-notes/exercises source .venv/bin/activate python llm-101/02/06_training_signal.py
passage: 46 tokens decoder-only, predict every next token : 45 training signals encoder-style, predict 15% hidden ones : 7 training signals ratio : 6.5x
Try: paste any paragraph from these notes into text and recount. Result: a 57-token paragraph gives 56 against 9, still 6.5×: the ratio is built in, every position against 15% of them.
Attention is a weighted sum, and addition doesn't depend on order, so attention on its own is blind to word order: "dog bites man" and "man bites dog" give "dog" exactly the same vector (the practice script shows this). Position has to be added to each token explicitly.
| Term | Meaning |
|---|---|
| positional encoding | Extra numbers added to each token's embedding to encode where it is in the text. |
| context window | The maximum number of tokens a model can attend over at once. Anything outside it doesn't exist for the model. |
The original transformer uses pairs of sine and cosine waves, each pair slower than the last, and adds the result to the embedding (concatenating would double its size):
PE(pos, 2i) = sin(pos / 10000^(2i/d)) PE(pos, 2i+1) = cos(pos / 10000^(2i/d)) pos starts at 0
import numpy as np
def positional_encoding(pos, d=8):
i = np.arange(d // 2)
angle = pos / 10000 ** (2 * i / d)
return np.ravel(np.column_stack([np.sin(angle), np.cos(angle)]))
for pos in (0, 1, 2, 50):
print(f"position {pos:3}:", np.round(positional_encoding(pos)[:4], 3))
position 0: [0. 1. 0. 1.] position 1: [0.841 0.54 0.1 0.995] position 2: [ 0.909 -0.416 0.199 0.98 ] position 50: [-0.262 0.965 -0.959 0.284]
The pairs have a useful property: moving from any position to the one k steps later is the same fixed rotation of each sine–cosine pair, wherever you start, so "k tokens apart" looks the same everywhere. Modern models build on this with RoPE (rotary position embedding), which rotates the query and key vectors by position instead of adding numbers to the embedding.
Every token attends to every token, so the attention pattern has length² entries and the work grows with the square of the length:
for ctx in (8_000, 16_000, 128_000):
print(f"{ctx:>7,} tokens: {ctx / 8_000:4.0f}x the text, {(ctx / 8_000) ** 2:5.0f}x the attention work")
8,000 tokens: 1x the text, 1x the attention work 16,000 tokens: 2x the text, 4x the attention work 128,000 tokens: 16x the text, 256x the attention work
That quadratic growth is why long context was hard for years, why long-context use is priced as it is, and why trimming what you send matters.
exercises/llm-101/02/07_position.py — runs attention on "dog bites man" and "man bites dog", with and without positional encoding.
without position dog/bites/man [1.74 0.55 1.16 0. ] man/bites/dog [1.74 0.55 1.16 0. ] with position dog/bites/man [2.01 1.3 1.06 0.99] man/bites/dog [2.59 0.34 0.93 0.98]
Try: make the position signal tiny: E = E + 0.01 * np.hstack(...). Result: at two decimals the two sentences look identical again; a position signal that is too weak is as good as none.
Generating runs the model once per token on everything written so far (§1). Without any reuse, the work grows quadratically with the length of the answer:
n = 1000
without_cache = n * (n + 1) // 2 # step 1 processes 1 token, step 2 two tokens, ...
with_cache = n # each token processed once
print(f"without caching: {without_cache:,} token-steps")
print(f"with caching: {with_cache:,} token-steps ({without_cache // with_cache}x less work)")
without caching: 500,500 token-steps with caching: 1,000 token-steps (500x less work)
When token 900 attends to token 3, the key and value vectors it needs from token 3 are the same ones token 899 needed; nothing about token 3 has changed. The KV cache stores each token's keys and values once instead of recomputing them at every step, turning quadratic work into linear work at the cost of memory.
| Term | Meaning |
|---|---|
| prefill | Processing the prompt. Done in parallel, so it is fast; this is the time to the first token. |
| decode | Generating each new token, one pass each. Serial, so it is the rest of the wait. |
per_token = 2 * 32 * 1024 * 2 # keys and values, 32 layers, 1024 wide, 2 bytes each
print(f"per token: {per_token // 1024} KiB")
for tokens in (1_000, 8_192, 128_000):
print(f"{tokens:>7,} tokens: {per_token * tokens / 1024**3:6.2f} GiB")
print(f"100 users at 8,192 tokens: {100 * per_token * 8_192 / 1024**3:.0f} GiB")
per token: 128 KiB 1,000 tokens: 0.12 GiB 8,192 tokens: 1.00 GiB 128,000 tokens: 15.62 GiB 100 users at 8,192 tokens: 100 GiB
Computed for an 8B-class model; sizes vary, but the shape doesn't.
This explains several things: long conversations get more expensive, not just slower; a server handles far fewer long chats at once than short ones; and the cache, not the model weights, usually limits how many users one GPU can serve. Providers price input and output tokens separately because prefill and decode are different operations on the hardware.
Providers also keep the cache for a prompt prefix that many requests share, such as a long system prompt, and charge less for reusing it. CampusX's text-to-SQL example (400 tokens in, 100 out, at the video's prices of $10 per million input and $50 per million output tokens):
full = 400 * 10 / 1e6 + 100 * 50 / 1e6
cached = 350 * 1 / 1e6 + 50 * 10 / 1e6 + 100 * 50 / 1e6 # 350 input tokens reused at $1 per million
print(f"per query: ${full:.5f} without caching, ${cached:.5f} with caching ({1 - cached / full:.0%} less)")
per query: $0.00900 without caching, $0.00585 with caching (35% less)
The first write costs more ($12.50 per million), and the cache expires after 5 minutes unless it's used again. It helps little for RAG, where most of the prompt changes with every request.
exercises/llm-101/02/08_kv_cache.py — generates 400 steps with and without a cache and compares the work and the time.
key rows computed, no cache : 80,200 key rows computed, cache : 400 time, no cache : 15.9 ms time, cache : 3.3 ms (5x faster) cache memory for an 8B-class model: 128 KiB per token, 1.00 GiB at 8,192 tokens
Try: set n = 800. Result: about 320,400 key computations against 800: the work saved doubles, and the timing gap widens with it.
The speed-up is much smaller than the work saved (200× less work, but only around 5–8× faster at 400 tokens, and it varies run to run) because Python's loop overhead dominates at this tiny size. In a real model each key costs far more, so the saving tracks the work more closely.
| Topic | In one line |
|---|---|
| RNNs | sequential, so they couldn't be trained in parallel at scale |
| Attention | every token takes a weighted mix of every other token |
| Self-attention | score (dot product) → weights (softmax, scaled by √d) → mix; Q, K, V are learned |
| Multi-head | several heads in parallel, each learning a different relationship |
| Model families | encoder sees everything, decoder looks left only (causal mask) |
| Decoder-only | most training signal per token, one interface for every task |
| Position | added explicitly; attention cost grows with context length squared |
| KV cache | reuse keys and values: linear instead of quadratic work, paid for in memory |