From text to token IDs and vectors

A word is not a fixed unit for a model

“Give apple the number 12345” is a useful starting picture. It leaves out two steps: deciding which pieces of text are tokens, and learning vectors for those pieces. A tokenizer turns text into a sequence that a particular model can accept.

A token can be a whole word, part of a word, punctuation or a piece involving whitespace. Case and spaces can matter: apple, Apple and apple need not have the same encoding. Some tokenizers normalize text; others preserve these distinctions. Hindi, accented text and emoji also depend on Unicode handling. One visible character is not guaranteed to be one token.

Source: Hugging Face: normalization and pre-tokenization ↗

Build pieces with a merge rule

Byte-pair encoding, or BPE, learns an ordered list of frequent adjacent-pair merges from training text. Encoding then applies the learned merge rules. It is one tokenizer family; it is not the neural model predicting an answer. Byte-level variants begin with byte symbols so that unfamiliar text can still be represented.

Source: Hugging Face: byte-pair encoding ↗

Try an invented merge list, already chosen for this exercise: a+p → ap, p+l → pl, ap+pl → appl, then appl+e → apple. Apply the rules in that order to the five starting pieces. This example specifies the rules; it does not claim these are a real model’s learned merges.

a p p l e → ap p l e → ap pl e → appl e → apple
Piece counts: 5 → 4 → 3 → 2 → 1.

Apply the lab’s ranked merges →

The ID is a label, not a meaning score

After splitting text, the vocabulary maps each token to an integer ID. Imagine that one invented vocabulary assigns apple → 12345. This is hypothetical, not a published mapping for a real model. Another vocabulary can give the same text a different ID—or split it into several tokens.

Source: Hugging Face Tokenizers: tokens and vocabulary IDs ↗

A bigger ID is not a bigger, better or more “apple-like” meaning. Adding two IDs does not combine their meanings. Consistently renaming IDs while rearranging the corresponding embedding rows and output-token mapping preserves the model’s behaviour. Changing only one lookup would break the agreement.

Original schematic; token boundary, ID and three-number vector are invented for this explanation. The lab uses its own small vocabulary.

Look up a row, then do arithmetic

An embedding matrix has one row per vocabulary token and D numerical coordinates per row. The ID selects its row. During neural training these entries can be learned; during ordinary generation this lookup uses the stored entries. The vector contains real-valued numbers, stored with finite numerical precision.

Source: PyTorch: embedding lookup ↗ · PyTorch: embedding matrices and learned representations ↗

For our invented row x = [0.2, −0.1, 0.4], try a learned-linear-operation miniature with chosen weights w = [1, 2, −1]. Multiply corresponding entries and add. These coordinates have no assigned labels such as sweetness or colour.

x · w = (0.2 × 1) + (−0.1 × 2) + (0.4 × −1) = −0.4
A matrix multiplication repeats such sums to produce several outputs.

The starting vector and the context differ

The same token initially retrieves the same embedding row in this model. Position information and later neural blocks then make its representation depend on the surrounding prefix. Models introduce order in different ways, including added position vectors or rotations used in attention.

Source: Su et al.: rotary position information ↗

Compare “I ate an apple” with “Apple makes computers”. First inspect their actual tokens; case alone may change them. Then distinguish a stored starting row from a context-dependent hidden vector. The lab’s hand-picked vectors let you inspect arithmetic; they were not trained to understand fruit or companies.

Inspect a small attention calculation →

Explain each change of representation

Trace a small example yourself

Write down the input, one intermediate calculation and the result before pressing the next control. Change one quantity and explain which later values should change.

Open the matching lab →

Sources and further reading

Original explanations, diagrams, invented numerical examples and answered exercises. Source papers and official documentation provide the underlying methods.