Interactive explainer

A language model is one loop, run over and over.

It turns text into numbers, mixes those numbers together so every word can see the words around it, and produces one guess about what comes next. Then it appends the guess and does the whole thing again. Everything else is detail — but the detail is where it gets interesting.

Start at step one Skip to the loop

One forward pass

01Tokenize Text becomes a list of integer IDs. 02Embed Each ID becomes a vector with meaning in it. 03Attend Every position pulls in context from earlier ones. 04Stack Repeat that block dozens of times. 05Predict One score for every token in the vocabulary. 06Sample Pick one, append it, start over.

Step 01 · Tokenization

The model never sees letters

Before anything else, your text is chopped into tokens — chunks that are usually a common word, a word fragment, or a single character. Each chunk is looked up in a fixed vocabulary and replaced by its integer ID. That list of integers is the only thing that enters the network.

The vocabulary is built once, before training, by a compression algorithm that merges the most frequent character pairs over and over. Common words survive whole; rare words get shattered. Type below and watch where the seams fall.

Tokenizer

Why this matters later. A model asked to count the letters in strawberry is not looking at letters. It sees a handful of opaque IDs. Ask the tokenizer above and you will see exactly what it is being handed — most of the surprising failures of language models start right here, at the seams.

Step 02 · Embeddings

Each ID becomes a direction in space

A token ID is just a label — ID 4021 is not "more" than ID 17. So the first layer is a lookup table: one learned vector per vocabulary entry. A real model uses a few thousand numbers per token. The toy below uses eight, each one hand-labelled so you can read them.

Nothing labels those axes in a real model. They emerge from training, and the useful fact about them is geometric: words used in similar contexts end up pointing in similar directions, and the directions themselves carry meaning you can do arithmetic with.

Embedding space

Position goes in here too. Attention has no built-in sense of order, so the model also injects where each token sits in the sequence — by rotating the vector by an angle proportional to its position. That rotation is what the "previous token" head in the next section is reading.

Step 03 · Self-attention

Every word gets to look at the other words

This is the part that made transformers work. Each position emits a query ("what am I looking for?"), and every position emits a key ("what am I?"). Dot the query against all the keys, soften the scores into percentages, and use them to average together the other positions' values.

Pick a query token below. The bars are real numbers: the scores are computed from actual query and key projections and pushed through the same softmax the model uses. Different heads learn to look for completely different things.

Attention head

The cost that shapes everything. Every token compares itself with every earlier token, so work grows with the square of the context length. Doubling the context roughly quadruples the attention cost. That single fact is why context windows are a headline feature and why so much research is aimed at this one matrix.

Step 04 · The stack

The same block, dozens of times over

One attention block plus one small feed-forward network makes a layer. Stack a few dozen of them and you have the model. Each layer writes its output back into a shared running vector — the residual stream — rather than replacing it, so early information survives all the way to the top.

Attention moves information between positions. The feed-forward network then works on each position alone, and is where most of the parameters live: it behaves like a big key–value memory of facts and patterns learned during training.

Residual stream

Step 05 · Logits and sampling

One number for every word it knows

At the top of the stack, the final vector is multiplied by the embedding table one more time, backwards. Out comes a logit for every entry in the vocabulary — a raw, unbounded score. Softmax turns those into probabilities that sum to one.

Then something has to choose. Always taking the top-scoring token makes the model repetitive; picking too freely makes it incoherent. The knobs below are the ones every chat interface exposes, and they all reshape this one distribution before the draw.

Next-token distribution

Temperature is a divisor, not a personality. Every logit is divided by it before the softmax. Below 1 the gaps widen and the top token dominates; above 1 they compress and the tail gets a real chance. At 0 it stops being a sample at all and becomes argmax.

Step 06 · The loop

Append and repeat

A model does not write a sentence. It writes one token, glues it to the end of the input, and runs the entire stack again from scratch. The fluency you read is that loop running a few hundred times, each pass knowing only what the passes before it happened to choose.

The generator below is a real, running language model — a small one trained by counting word sequences in the corpus at the bottom of this page. It is far too simple to be useful, and the loop it runs is exactly the loop a frontier model runs.

Autoregressive generation

Why the first token is the slow one. The whole prompt has to be processed before anything appears, and every later token re-reads all of it. Models cache the keys and values they already computed so each new step only does work for the one new position — which is why output streams out at a steady pace after an initial pause.

Step 07 · Training

Where the numbers came from

Every weight starts as noise. Show the model a piece of real text, hide the next token, and measure how surprised it is by the truth — that surprise is the loss. Nudge every weight slightly in the direction that would have reduced it. Repeat a few trillion times.

The demo below is not an animation. It is gradient descent running in your browser on a real model: a softmax over character pairs, roughly a thousand weights, trained on the corpus at the bottom of the page. Press train and watch the loss fall and the weight matrix organise itself.

Gradient descent, live

What the full recipe adds

Pretraining. The same next-token objective as the demo, on a very large text corpus. This is where almost all the knowledge and nearly all the compute goes.

Supervised fine-tuning. Further training on examples of instructions followed well, which is what turns a text continuer into something that answers you.

Preference tuning. Humans (or a model standing in for them) rank competing answers; the model is optimised toward the preferred ones. Tone, refusals and helpfulness are largely shaped here.

Everything at inference. System prompts, retrieved documents and tool results are not training. They are just more tokens placed in the context before the loop starts.

Step 08 · Consequences

What the mechanism explains

Most well-known quirks of language models are not bolted-on flaws. They fall straight out of the seven steps above.

Behaviour, and where it comes from

What you see	Where it comes from
Confident wrong answers	The output is a probability distribution over plausible continuations. Nothing in the objective rewards saying "I don't know" unless training taught it to.
Can't reliably count letters	Step 01. Words arrive as a handful of opaque IDs; the spelling was discarded before the network saw anything.
Arithmetic that looks almost right	Numbers are tokenized into fragments, and each digit of the answer is produced by the same next-token guess as everything else.
Forgets earlier conversation	There is no memory between calls. Anything the model "remembers" was re-sent as tokens; past the context window it is not sent at all.
Rambles or repeats itself	Step 06. Low temperature narrows the loop onto its own highest-probability groove; a repeated phrase makes itself more likely next time.
Same question, different answer	Sampling is a draw, not a lookup. Change the seed or the temperature and you change the text.
Thinking out loud helps it	Each token gets a fixed amount of computation. Writing intermediate steps buys more forward passes, and puts its own reasoning back in the context.
Knowledge stops at a date	Weights are frozen after training. Anything newer has to arrive through the context — search results, files, tool output.

Glossary

Term	Meaning
Token	A chunk of text with an ID in the model's fixed vocabulary. Roughly ¾ of an English word on average.
Embedding	The learned vector a token ID is replaced by, and the space those vectors live in.
Context window	The maximum number of tokens the model can attend over in one pass. Prompt and output share it.
Head	One independent query/key/value projection inside an attention layer. A layer runs many in parallel.
Residual stream	The running vector each layer reads from and adds to, rather than overwriting.
Logit	A raw, unnormalised score for one vocabulary entry, before softmax turns the set of them into probabilities.
Temperature	A divisor applied to logits before softmax. Lower is sharper and more deterministic.
Top-k / top-p	Truncation rules applied before the draw: keep the k best tokens, or the smallest set whose probabilities reach p.
Perplexity	The exponential of the average loss. Read it as "how many equally likely options the model felt it was choosing between".
Parameter	One learned number. A modern model has hundreds of billions of them, and all of them are just floats.
KV cache	Stored keys and values from earlier positions, so each generation step only computes the new one.
Inference	Running the trained model. Nothing is learned; no weight changes.

The corpus Sections 6 and 7 are trained on exactly this text.