This is a conceptual path for autoregressive transformer generation, not a complete diagram of every architecture. The three experiments below are independent teaching toys; they do not combine into a trained LLM.
Start with an exact text string. Case and spaces can affect tokenization.
Type English, Hindi or a mixture. This reversible, character-based teaching tokenizer repeatedly applies a fixed, ranked list of adjacent merges. It preserves case, spaces and Unicode code points. Its tiny vocabulary and IDs are our own examples, unrelated to any actual GPT or other provider.
At most 160 Unicode code points. A token can include a space or only part of a word. The symbol ␠ makes a space visible.
Example: I | ␠eat | ␠an | ␠apple | .
I · 1073␠eat · 56␠an · 57␠apple · 58. · 1046
At position 0, I selects ID 1073 and the hand-picked row [0.2, -0.1, 0.6]. Position is a separate input.
A few rows in the hand-picked embedding illustration
Piece
ID label
Toy vector [x1, x2, x3]
I
1073
[0.2, -0.1, 0.6]
␠an
57
[0.1, 0.5, 0.3]
apple
7
[0.7, 0.1, -0.3]
␠apple
58
[0.8, 0.2, -0.4]
Apple
15
[-0.4, 0.7, 0.1]
These axes have no assigned semantic meaning. The vector values were hand-picked, not trained or computed from the numeric ID. Unsupported pieces show “no row” in this small illustration. We add a toy position vector [0.1 × position, 0, 0], starting at position 0; this is our illustrative rule, not a claim about a provider’s position encoding.
Edit the two query coordinates. The three key/value pairs stay fixed. Every number here is hand-picked; the query is not derived from your text. This is one scaled dot-product attention calculation, without learned projections, masking, multiple heads or other neural layers. It is not a trained transformer or a view of a model’s hidden reasoning.
Each coordinate: -3 to 3. Dimension d = 2. Score = (q1 × k1 + q2 × k2) / √2.
Weights sum to 1. The result is a mixture of values, not a selected word or an answer probability.
Try [0, 0]: all scores are zero, so each weight is 1/3 and the weighted value is [1, 1]. Try [0, 3]: the second and third keys receive larger scores than the first. Attention weights describe this operator’s mixing; they do not by themselves explain an entire model’s reasoning.
This working bigram model learns a table of observed adjacent-token counts. It uses the same tiny tokenizer, but does not use the embedding or attention demonstrations above. It looks only one token back. LLMs instead learn neural weights and can condition on much longer contexts; pair counting is not how this page claims to train an LLM.
1–12 nonempty lines; ≤160 Unicode code points per line, ≤1,600 total. Each line gets one EOS marker (ID 0). No transitions cross line boundaries; blank lines are ignored. Counts have no smoothing, so unobserved transitions have zero probability.
In the sample corpus, ␠an is followed by ␠apple twice and ␠orange once.
Prompt ≤160 code points. Seed: integer 0–4294967295. The same trained counts, prompt, temperature and seed reproduce the same run. Changing prompt, temperature or seed resets it. At most 12 new tokens; EOS stops sooner.
The complete prompt plus generated text must also stay within 160 Unicode code points and 161 token IDs, including EOS. A piece that would exceed either limit is not appended; shorten the prompt and reset.
Previous token:
Next-token candidates from the current count row
Candidate / ID
Count
ln(count/total)
At T=1
Chosen T
“␠apple” · 58
2
-0.4055
66.67%
66.67%
“␠orange” · 59
1
-1.0986
33.33%
33.33%
For positive T, use softmax(log(count/total) / T) over observed candidates. At T = 1 this recovers the count proportions. Lower T sharpens this row; higher T flattens it. At T = 0 we choose the largest count directly, breaking ties by the smaller ID. Temperature never gives an unobserved transition a nonzero chance here.
I eat an
Ready for the first choice. Each tap appends one token, then displays the new previous token’s candidates.
At greedy temperature, “I eat an” continues to “I eat an apple.”, then EOS. “Apple makes” has two equally counted continuations: ␠phones and ␠computers. Try a seed or edit the corpus and retrain. If a previous token has no observed successors, the toy stops and says so rather than inventing a fallback.
Everything in these experiments runs in this browser. There is no external model, API call, account, upload or saved model state. Reloading restores the examples. The toy’s probabilities measure its tiny corpus, not truth, intelligence or a commercial model’s capabilities.
