LLM 101 · Section 1. What a language model outputs, how it is trained, why writing steps helps it reason, what scale changes, what a foundation model is, and the two kinds of model used in real systems.
A language model takes some text and returns the probability of every possible next token. That is the whole definition. Transformers, training and GPUs are all machinery for making that one output more accurate.
Given the text "The capital of France is", the model doesn't return a sentence. It returns one probability for every token in its vocabulary:
" Paris" 0.87
" located" 0.04
" a" 0.02
" home" 0.01
" known" 0.01
… 100,272 more rows, each a tiny number
─────
every row adds to 1.00
The probabilities are illustrative; they differ per model. The shape of the output is always this.
| Term | Meaning |
|---|---|
| token | A word-piece, not a word. " Paris" is one token; "thermodynamics" is three (ther + m + odynamics). |
| vocabulary | The fixed list of tokens a model knows: 100,277 for GPT-4's tokenizer, 50,257 for GPT-2. |
| probability distribution | The whole table: every token with its probability. The values always add up to 1. |
| logit | The raw score the network gives each token before it becomes a probability. Any number, even negative. |
| softmax | The step that turns logits into probabilities: raise e to each score, then divide each by the total. |
Text is produced one token at a time. At each step the model outputs a table, one token is picked from it and appended, and the longer text goes back into the model:
| Step | Text going in | Top of the table | Picked |
|---|---|---|---|
| 1 | The capital of France is | " Paris" 0.87 | " Paris" |
| 2 | …France is Paris | "." 0.61 | "." |
| 3 | …France is Paris. | " It" 0.22 | " It" |
The model never writes a sentence directly. It produces one table per token, and a separate step, the sampler, picks a row from each.
The network first gives every token a logit, then softmax turns the logits into the table. With a three-token vocabulary:
import math
logits = {" Paris": 5.0, " located": 2.0, " a": 1.2}
total = sum(math.exp(v) for v in logits.values())
for token, v in logits.items():
print(f"{token!r:12} {math.exp(v) / total:.3f}")
' Paris' 0.933 ' located' 0.046 ' a' 0.021
" Paris" at 0.933 wins about 93 times in 100. Chat models usually sample.import math
logits = [5.0, 2.0, 1.2]
for t in (0.5, 1, 2):
exps = [math.exp(v / t) for v in logits]
print(f"temperature {t}:", " / ".join(f"{x / sum(exps):.4f}" for x in exps))
temperature 0.5: 0.9970 / 0.0025 / 0.0005 temperature 1: 0.9327 / 0.0464 / 0.0209 temperature 2: 0.7285 / 0.1625 / 0.1090
This is why the same prompt can give different answers. Inference settings are covered in their own section (§5).
exercises/llm-101/01/01_next_token_table.py — prints the real probability table GPT-2 produces for "The capital of France is".
# Section 01: see the table a language model actually outputs.
# Predict first: what will the top next token be after "The capital of France is"?
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("gpt2", clean_up_tokenization_spaces=True)
model = AutoModelForCausalLM.from_pretrained("gpt2")
text = "The capital of France is"
ids = tok(text, return_tensors="pt").input_ids
with torch.no_grad():
scores = model(ids).logits[0, -1] # one score for every possible next token
table = torch.softmax(scores, dim=-1) # scores -> probabilities that add to 1
print(f"The table has {len(table):,} rows and they add up to {table.sum():.4f}\n")
top = torch.topk(table, 8)
for p, i in zip(top.values, top.indices):
print(f" {tok.decode(i)!r:14} {p:.3f}")
# Then try: change `text` to "The capital of Japan is", then to "I love eating".
This one needs torch, transformers, which can't run in a browser. Run it locally:
cd ~/Downloads/ai-engineer-notes/exercises source .venv/bin/activate python llm-101/01/01_next_token_table.py
The table has 50,257 rows and they add up to 1.0000 ' the' 0.085 ' now' 0.048 ' a' 0.046 ' France' 0.032 ' Paris' 0.032 ' in' 0.027 ' also' 0.026 ' not' 0.024
Try: change the text to "The capital of Japan is", then to "I love eating". Result: Japan gives ' the' 0.094 then ' Tokyo' 0.067; "I love eating" spreads thinly (',' 0.064, ' and' 0.060), because many continuations are reasonable.
GPT-2 is a small 2019 model, which is why " Paris" comes fifth at 0.032 in the real output. A modern large model puts most of its probability on it.
A language model learns by predicting the next token of real text, over and over. Take a sentence, hide the next token, let the model guess, and score how much probability it gave to the true token. Then adjust the model so that probability goes up.
The score is the loss: -log(probability given to the true token). Lower is better, and zero means the model was certain and correct. For the sentence "The cat sat on the mat" with the last token hidden:
import math
for p in (0.95, 0.60, 0.01):
print(f"model gave ' mat' {p:.2f} -> loss {-math.log(p):.2f}")
model gave ' mat' 0.95 -> loss 0.05 model gave ' mat' 0.60 -> loss 0.51 model gave ' mat' 0.01 -> loss 4.61
The penalty isn't linear. Going from 0.95 to 0.60 costs 0.46, but going from 0.60 to 0.01 costs 4.10. Training punishes being confident and wrong far more than being unsure.
| Term | Meaning |
|---|---|
| loss (cross-entropy) | How wrong one guess was: -log p of the true token. Called categorical_crossentropy in Keras. |
| training | Repeating the guess-and-score step and nudging the model's parameters so the true tokens get more probability. Real training updates once per batch (about a million tokens in Karpathy's run) and reports the average loss. |
| self-supervised | Supervised learning where the labels come from the text itself: the next token is already in the sentence, so no human labelling is needed. |
Every position in a sentence is used, so a sentence of n tokens gives n − 1 examples. CampusX's example, split by word for readability:
"Hi my name is Nitish" → 4 training examples Hi → my Hi my → name Hi my name → is Hi my name is → Nitish
Token IDs are labels, not quantities: token 4 isn't "bigger" than token 2, and a predicted 2.7 would mean nothing. So next-token prediction is a classification problem over the whole vocabulary, with one probability per token.
The probabilities always add up to 1, so something always gets the probability, even when the model has no reliable knowledge. There is no "I don't know" row. A made-up answer is the same mechanism working normally on a question it has nothing solid for.
-log p, adjust. At every position, over trillions of tokens (FineWeb, used by Karpathy, is 15 trillion).The CampusX video calls this "supervised learning" and splits text by whole words; LLMs split it into tokens. The idea is the same.
exercises/llm-101/01/02_loss.py — prints the loss GPT-2 assigns to every token of two sentences.
# Section 02: score how surprised a model is by each real next token.
# Predict first: which sentence gets the higher total loss, and at which word?
import math
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("gpt2", clean_up_tokenization_spaces=True)
model = AutoModelForCausalLM.from_pretrained("gpt2")
for text in ["The cat sat on the mat.", "The cat sat on the thermodynamics."]:
ids = tok(text, return_tensors="pt").input_ids[0]
with torch.no_grad():
tables = torch.softmax(model(ids[None]).logits[0], dim=-1)
print(text)
total = 0.0
for pos in range(1, len(ids)):
p = tables[pos - 1, ids[pos]].item() # probability given to the real next token
loss = -math.log(p)
total += loss
print(f" {tok.decode(ids[pos])!r:18} p = {p:.4f} loss = {loss:.2f}")
print(f" total loss {total:.2f}\n")
# Then try: add a sentence that is true but unusual, e.g. "Paris is the capital of France."
This one needs torch, transformers, which can't run in a browser. Run it locally:
cd ~/Downloads/ai-engineer-notes/exercises source .venv/bin/activate python llm-101/01/02_loss.py
The cat sat on the mat. ' cat' p = 0.0001 loss = 9.20 ' sat' p = 0.0007 loss = 7.26 ' on' p = 0.2240 loss = 1.50 ' the' p = 0.4537 loss = 0.79 ' mat' p = 0.0040 loss = 5.51 '.' p = 0.0631 loss = 2.76 total loss 27.02 The cat sat on the thermodynamics. ' cat' p = 0.0001 loss = 9.20 ' sat' p = 0.0007 loss = 7.26 ' on' p = 0.2240 loss = 1.50 ' the' p = 0.4537 loss = 0.79 ' therm' p = 0.0000 loss = 11.47 'odynamics' p = 0.0000 loss = 10.82 '.' p = 0.0026 loss = 5.96 total loss 46.99
Try: replace the two sentences with "Paris is the capital of France." Result: the total is 16.56; the cheapest token is ' of' after "capital" (p = 0.72, loss 0.32).
The two sentences cost the same until the last word: 27.02 in total for "mat" against 46.99 for "thermodynamics", which GPT-2 splits into two unexpected tokens (11.47 + 10.82).
Producing one token takes one forward pass through the network, and a forward pass does a fixed amount of computation, because the network has a fixed number of layers (about a hundred in a modern model). The model can't "think harder" inside one pass. The only way it gets more computation is to produce more tokens.
A. "17 × 24 = "
→ one token, one pass. Often wrong.
B. "17 × 24 = ? Work step by step."
→ "17 × 20 = 340" ← a pass
"17 × 4 = 68" ← a pass
"340 + 68 = 408" ← a pass
"The answer is 408" ← a pass
In B, every written token is fed back in as input, so "17 × 20 = 340" becomes text the model can read while computing the next token. It stores partial results on the page and builds on them.
If the model writes "The answer is 408, because 17 × 20 = 340…", the 408 was produced in one pass before any working existed. The explanation after it can't have helped. Karpathy calls this post-hoc justification. Working must come before the answer.
Each written step is itself one pass and can be wrong. When exactness matters, the better route is a tool: Karpathy has the model write and run code instead. Asked to count a row of dots, the model guessed 161; Python's .count() gave the true 177.
exercises/llm-101/01/03_passes.py — counts how many passes each answer style gives the model.
# Section 03: a model gets one pass of computation per token it writes.
# Predict first: how many passes does each answer buy the model?
import tiktoken
enc = tiktoken.get_encoding("cl100k_base")
answers = {
"straight to it": "408",
"step by step": "17 × 20 = 340. 17 × 4 = 68. 340 + 68 = 408. The answer is 408.",
}
for name, answer in answers.items():
n = len(enc.encode(answer))
print(f"{name:<15} {n:>3} tokens -> {n} passes")
# Then try: write your own step-by-step working for 23 × 47 and count its passes.
This one needs tiktoken, which can't run in a browser. Run it locally:
cd ~/Downloads/ai-engineer-notes/exercises source .venv/bin/activate python llm-101/01/03_passes.py
straight to it 1 tokens -> 1 passes step by step 32 tokens -> 32 passes
Try: write your own step-by-step working for 23 × 47 and count its passes. Result: 23 × 40 = 920. 23 × 7 = 161. 920 + 161 = 1081. The answer is 1081. is 34 tokens, so 34 passes.
Bigger models trained on more data get better, and part of that improvement is very predictable while part of it looks sudden.
| Term | Meaning |
|---|---|
| parameters | The internal numbers adjusted during training. "7 billion" means seven billion of them. |
| scaling law | The observed fact that loss falls smoothly and predictably as two things grow: parameters (N) and training data (D). Predictable enough to forecast a model before building it. |
| emergence | An ability that seems absent in smaller models and present in larger ones, appearing as a jump rather than gradually. |
Train the same design at four sizes and measure the average loss and 5-digit addition accuracy:
size average loss 5-digit addition ────────────────────────────────────────────── 1 billion 3.2 0% 7 billion 2.9 0% 30 billion 2.6 0% 150 billion 2.3 61%
Illustrative numbers with the real shape.
The loss column improves steadily, but addition jumps. One reason: 5-digit addition is scored right or wrong. Four correct digits out of five still scores zero. If each digit is right with probability p, the whole answer is right with probability p5, which stays near zero for a long time and then rises quickly. Much of the "cliff" is in the way the test is scored, not in the model.
This is the argument of Schaeffer et al. (2023): with a score that gives partial credit, or more test questions, many claimed emergent abilities become smooth curves. It's a strong case but not settled; the paper itself says emergence "may not be" a fundamental property of scale.
exercises/llm-101/01/04_emergence.py — shows smooth per-digit accuracy turning into a sudden jump in whole-answer accuracy.
size each digit whole answer 1B 0.30 0.2% 3B 0.45 1.8% 7B 0.60 7.8% 30B 0.75 23.7% 70B 0.85 44.4% 150B 0.93 69.6% 400B 0.98 90.4%
Try: set digits = 10. Result: at 0.85 per digit the whole answer is right only 19.7% of the time, and the jump moves to larger sizes (48.4% at 150B, 81.7% at 400B).
The sizes and per-digit accuracies are made up to show the shape; the arithmetic (p to the power of the number of digits) is exact.
A foundation model is a large model pre-trained on huge, varied data with a general task (for text, next-token prediction), whose learning transfers to many other jobs. A task-specific model is trained for one job only.
| Task-specific model | Foundation model | |
|---|---|---|
| How you get it | Collect ~20,000 labelled reviews, train a model | Prompt a general model, or fine-tune it on a few hundred examples |
| Data needed | A lot of labelled data | None (prompting) or a little (fine-tuning) |
| Speed per call | ~3 ms | ~800 ms |
| Cost per call | very low | hundreds of times higher |
| Other jobs | none; a new category means starting over | also summarises, translates, writes code |
| Term | Meaning |
|---|---|
| pre-training | The expensive first stage: training on a huge dataset with a general task. |
| alignment | Adjusting the model's behaviour, e.g. people rank its answers and it is rewarded for the best ones. |
| fine-tuning | Further training on a small labelled dataset for a specific job. CampusX's analogy: engineering college is pre-training, a company's induction programme is fine-tuning. |
Building a good task-specific model needs data, money and a team that can train models from scratch. CampusX's point is that most companies lacked these, so their models were mediocre. A foundation model arrives already strong because someone else paid for pre-training and alignment; you only adapt it by prompting or fine-tuning.
A common sequence: use a foundation model to find out whether the idea is worth building, and build a small specialised model only once it is and the volume makes cost matter.
exercises/llm-101/01/05_task_specific.py — builds a tiny task-specific sentiment model from labelled examples.
'great acting, wonderful' -> pos 'boring and awful' -> neg 'an absolute masterpiece' -> no idea 'summarise this review for me' -> no idea 4 predictions in 11 microseconds
Try: add ("a masterpiece", "pos") to the training data. Result: "an absolute masterpiece" becomes pos, but "summarise this review" is still no idea: more labels fix gaps, not the fact that it does one job.
There are two kinds of model in most AI applications, and they return different things for the same input.
| Generative model | Representation (embedding) model | |
|---|---|---|
| Returns | text, one token at a time | a fixed-length list of numbers (an embedding) |
| Example output for "How do I reset my password?" | "Go to Settings, then click Reset Password…" | [0.04, -0.21, 0.88, …], always the same length |
| Used for | chat, writing, answering | search by meaning, clustering, classification |
| Size | large (GPT-2 small: 124 million parameters; chat models: billions) | small (the exercise's model: 22.7 million) |
| LangChain name | language model (LLM / chat model) | embedding model |
An embedding is a position in space that stands for the meaning of the input. On its own it means nothing; the point is that texts with similar meanings get similar embeddings. Closeness is measured with cosine similarity, the cosine of the angle between two vectors: 1 means same direction, 0 means unrelated.
import math
def cosine(a, b):
dot = sum(x * y for x, y in zip(a, b))
return dot / (math.hypot(*a) * math.hypot(*b))
question = [0.9, 0.1, 0.3] # made-up 3-number embeddings
login = [0.8, 0.2, 0.4]
hours = [0.1, 0.9, 0.2]
print(f"question vs login: {cosine(question, login):.3f}")
print(f"question vs hours: {cosine(question, hours):.3f}")
question vs login: 0.984 question vs hours: 0.271
The vectors here are made up to show the calculation; real embeddings have hundreds of numbers. The practice script uses a real model.
Search that works on meaning (semantic search) embeds every document once and stores the vectors in a vector database. Each question is embedded, and the closest documents are found by cosine similarity (CampusX calls this step retrieve). The generative model then reads only those few documents and writes the answer. This is the basis of RAG (retrieval-augmented generation).
exercises/llm-101/01/06_embeddings.py — embeds three sentences with a real model and compares them to a question.
# Section 06: turn sentences into numbers and compare them by meaning.
# Predict first: which sentence is closest to "How do I reset my password?"
import numpy as np
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("all-MiniLM-L6-v2", tokenizer_kwargs={"clean_up_tokenization_spaces": True})
sentences = ["How do I reset my password?", "I forgot my login", "What time do you close?"]
vectors = model.encode(sentences, normalize_embeddings=True)
print(f"Each sentence becomes {vectors.shape[1]} numbers.")
print("First five for sentence one:", np.round(vectors[0][:5], 3), "\n")
for sentence, v in zip(sentences, vectors):
print(f" {vectors[0] @ v:.3f} {sentence}") # 1.0 means identical meaning
# Then try: add "Can you help me get back into my account?". Where does it rank?
This one needs sentence_transformers, which can't run in a browser. Run it locally:
cd ~/Downloads/ai-engineer-notes/exercises source .venv/bin/activate python llm-101/01/06_embeddings.py
Each sentence becomes 384 numbers. First five for sentence one: [ 0.007 -0.063 -0.072 -0.032 -0.043] 1.000 How do I reset my password? 0.611 I forgot my login 0.122 What time do you close?
Try: add "Can you help me get back into my account?" to the list. Result: it scores 0.565, close to "I forgot my login" and far above the closing-time question: similar meaning, different words.
| Idea | In one line |
|---|---|
| Language model | text in, a probability for every next token out |
| Generation | one token per step; a sampler picks from each table |
| Training | predict the next token, score with -log p, adjust; the text labels itself |
| Reasoning | each token is one fixed-size pass; writing steps gives more passes |
| Scaling | loss improves smoothly; pass-or-fail tasks can look like sudden jumps |
| Foundation model | general pre-training that transfers; adapt by prompting or fine-tuning |
| Generative vs representation | text out vs meaning-vectors out; real systems use both |