How training becomes token-by-token generation

Training changes parameters; generation uses them

A next-token training example supplies a prefix and the token that actually follows it in the data. The model predicts a distribution. A loss measures how poorly it predicted the target. Backpropagation computes gradients, and an optimizer adjusts learned parameters, including matrices and embeddings, across many examples.

Source: PyTorch: loss, gradients and parameter updates ↗

For one target token, loss = −ln(p(target)).
If p = 0.5, loss ≈ 0.693; if p = 0.25, loss ≈ 1.386. Giving the observed target less probability costs more.

Source: PyTorch: cross-entropy loss ↗

Training can score many positions in parallel while a causal mask prevents a position from seeing later target tokens. In ordinary inference, a new prompt changes the context, not the stored model weights. Later post-training can use instruction demonstrations and human preference comparisons to change behaviour; it is a further learning stage.

Source: Radford et al.: a Transformer language model ↗ · Ouyang et al.: instruction and preference training ↗

What the neural blocks calculate

In a common causal Transformer, learned matrix multiplications produce query, key and value vectors. Attention compares queries with allowed keys, normalizes the scores and mixes value vectors. The causal restriction lets a position use its prefix, including itself, while excluding future tokens.

Source: Vaswani et al.: Attention Is All You Need ↗

A feed-forward network applies learned linear operations and a nonlinearity at each position. Residual paths add a block’s contribution back to its input; normalization controls activation scales. Repeated blocks transform representations. Exact ordering and position methods vary across architectures. This is a simplified decoder-only account, not every Transformer’s complete diagram.

Source: Radford et al.: a Transformer language model ↗

Hand-picked attention miniature: weights [0.75, 0.25], values [2, 0] and [0, 4].
0.75[2, 0] + 0.25[0, 4] = [1.5, 1]. The weights sum to 1. This arithmetic is chosen, not a trained language model.

Change query, keys and values in the lab →

Turn scores into a distribution

The final representation is projected to one score, called a logit, per vocabulary token. Softmax exponentiates the logits and divides by their total. The resulting positive probabilities sum to 1. These scores are calculated from vectors and learned weights, not from the sizes of token IDs.

Source: PyTorch: softmax ↗

An invented three-token softmax
Candidate	Logit z	exp(z)	Probability
apple	ln(2) ≈ 0.693	2	2/4 = 0.50
pear	0	1	1/4 = 0.25
EOS · end	0	1	1/4 = 0.25

Check the denominator: 2 + 1 + 1 = 4. A probability of 0.50 means half the sampling mass in this invented distribution, not a score of “half an apple”. A different prefix can change every logit. Real vocabularies contain many more candidates.

Choose, append, repeat

Greedy decoding chooses the highest-probability token. Sampling can choose another candidate: pear remains possible in the table above. After appending the chosen token, calculate the next distribution from the extended prefix and repeat. An end token or generation limit stops the loop. Decoding settings change selection; they do not retrain the model.

Source: Hugging Face: generation strategies ↗

Original simplified decoder-only generation flow. The arrow returns from “append” to the current prefix; an end token (EOS) signals completion.

The lab’s actual bigram count model conditions only on the preceding token. In its fixed sample, the token for an precedes apple twice and orange once: at temperature 1, probabilities are 2/3 and 1/3. No smoothing adds unseen transitions; an unseen previous token stops. This count table has no neural embeddings, attention or gradient training.

Source: Stanford: bigram language models ↗

Train counts, then generate step by step →

Likelihood does not establish truth

A likely continuation can repeat a common misconception or invent a convincing detail. Next-token probability is not the probability that a whole answer is true. Even a numerical confidence estimate needs calibration against checked outcomes; softmax alone does not provide that guarantee.

Source: Lin et al.: TruthfulQA ↗ · Guo et al.: probability calibration ↗

For a factual claim, identify what observation, calculation or original source could check it. For generated arithmetic, recompute a step. For a citation, open the source and match the claim. This adds evidence to the language pattern.

Practise checking AI’s work →

Keep learning and choosing separate

Trace a small example yourself

Write down the input, one intermediate calculation and the result before pressing the next control. Change one quantity and explain which later values should change.

Open the matching lab →

Sources and further reading

Original explanations, diagrams, invented numerical examples and answered exercises. Source papers and official documentation provide the underlying methods.