--- title: "Anatomy of an LLM: tokens and embeddings" date: 2026-06-11 description: "A network does not see letters - it sees vectors. How text is split into tokens, why not into words nor characters, how subword algorithms (BPE, WordPiece, Unigram) build a vocabulary, and how the famous geometry of meaning is born, where \"king - man + woman β‰ˆ queen\" - together with an honest critique of that analogy, and with the positional encoding that primers leave out." tags: ["llm anatomy", "ai", "llm", "nlp", "deep-dive"] --- ## πŸš€ Intro In the [previous stop of this series](/en/posts/llm-sieci-parametry-trening) - one of the branches of the [overview post "Anatomy of an LLM"](/en/posts/anatomia-llm) - we assembled a whole network from neurons and showed how training turns random noise into a model that understands language. All along, though, we kept talking about "processing tokens" and "predicting the next token," while passing over the most basic question: **what actually is a token, and how does a word become a number** that can be multiplied? Because the network does not see letters. It does not see words. It sees **vectors** - sequences of numbers. Before any of the matrix multiplications from the previous post can even begin, the text has to go through two stages: first it is **cut into pieces** (tokens), and then each piece is given a **vector** (an embedding). This is the bridge between the world of characters and the world of numbers - and it has a surprisingly rich construction. In this post we will see why modern models cut text neither into words nor into single characters, but into something in between. We will trace how that famous **geometry of meaning** is born, where directions in space correspond to concepts and subtracting vectors yields analogies. And - in the spirit of the whole series - we will look at it critically: because the example "king - man + woman β‰ˆ queen" is a beautiful intuition, but one that is often over-interpreted. Finally we will touch on something primers usually skip: how the model knows in what **order** the words came at all. > πŸ’‘ **How to read this.** The main thread is descriptive and self-contained. I tuck the formulas into expandable "for the curious" blocks - if you like math, open them; if not, skip them with no loss to understanding. Every claim has a footnote to its source paper at the end. ## πŸ“‹ TL;DR - **A token is not a word.** Splitting into whole words blows up the vocabulary (especially in inflected languages like Polish) and trips over unknown words; splitting into characters makes sequences hopelessly long. The standard is **subword tokenization** - common words become a single token, rare ones split into known pieces. - **Subword algorithms:** **BPE** (merges the most frequent pairs - the core of GPT), **WordPiece** (a probabilistic criterion - BERT), **Unigram/SentencePiece** (prunes a large vocabulary - language-independent). - **How many tokens?** The vocabularies of today's models range from about 50k to about 200k tokens. Rule of thumb: about 4 characters per token in English - much fewer in inflected languages, so the same text "costs" more tokens. - **An embedding is a vector of meaning.** Each token gets a row in a trained **embedding matrix** $E$. The result is a space where closeness = similarity of meaning, and where directions can be interpretable. - **The famous analogy, with a critical eye.** "King - man + woman β‰ˆ queen" shows the linear structure of the space, but it does **not prove** that the model "understands" relations - it is partly an artifact of the metric and the choice of examples (Linzen; Drozd et al.). - **Static vs contextual.** word2vec/GloVe give one vector per word (the same "bat" in "vampire bat" and "baseball bat"). Transformers give **contextual** representations - the vector depends on the sentence. - **Position.** Attention by itself does not know the order of tokens - it has to be **injected**: sinusoidal encoding, learned tables, and today most often **RoPE** (rotation) and **ALiBi** (a linear penalty for distance). πŸ‘‰ **Play with it:** on the interactive page you have a [section on tokens and embeddings](/en/llm/anatomy#tokeny) - watch how text breaks into pieces and how the vectors arrange themselves in the space of meaning. ## βœ‚οΈ Why neither words nor letters It would seem most natural to split text into **words**. The trouble is that language is "open" - new words, proper names, blends and typos keep appearing. A vocabulary has to be finite, so anything not in it lands in a single "unknown" bin (the `` token, from *unknown*). This is the so-called **OOV** problem (*out-of-vocabulary*). In **inflected** languages - like Polish, Finnish or Turkish - it hurts twice over. A single word has dozens of forms (Polish *kot, kota, kotu, kotem, kotΓ³w, kotami...* - "cat" in various cases). If each form were a separate token, the vocabulary would explode, and the rarer forms would still be unknown to the model. Polish is practically a textbook adversary of the "word = token" approach. The other extreme is splitting into single **characters**. OOV vanishes (the alphabet is small and closed), but a different problem appears: sequences become **many times longer**. A word like "running" is one or two subword tokens, but seven characters. And the cost of the attention mechanism grows with the **square** of the sequence length (we will get to that in the next post) - so longer sequences mean markedly more expensive computation. On top of that, the whole burden of composing characters into meaning then falls on the network itself. **Subword tokenization** is the golden mean: common words stay as one token, while rare and unknown ones break into smaller, known pieces. The vocabulary has a controlled size, OOV practically disappears, and sequences stay short. This is today's standard across all large language models. ## 🧩 Subword algorithms: BPE, WordPiece, Unigram How does one actually decide **which** pieces should be tokens? This is where three families of algorithms come in. **Byte-Pair Encoding (BPE).** Curiously, it was born as a **data compression** algorithm - Philip Gage described it in 1994[^gage]. The idea is dead simple: find the most frequently adjacent pair of symbols and replace it with one new symbol; repeat. Sennrich, Haddow and Birch (2016) carried this idea over to language processing[^sennrich]: we start from single characters and in each step **merge** the most frequently co-occurring pair, until the vocabulary reaches a target size. The list of those merges becomes the recipe for tokenizing any new text. GPT-2 added a clever variant: **byte-level BPE**, operating on UTF-8 **bytes** instead of characters[^gpt2]. Since every possible byte (there are 256) is in the base alphabet, **any** Unicode string can be encoded without `` - zero OOV by construction. This approach reigns in the GPT family. **WordPiece.** Google's variant (Schuster and Nakajima, 2012)[^schuster], popularized by their translation system[^gnmt] and used in **BERT**[^bert]. It differs from BPE in its merge criterion: instead of plain "most frequent pair," WordPiece merges the pair that **most increases the probability** of the corpus under a simple statistical model. A probabilistic criterion, not a purely frequency-based one. **Unigram LM and SentencePiece.** Kudo (2018) approached it from the other end[^kudo-unigram]: instead of building the vocabulary bottom-up by merging, it starts from a **large** set of candidates and **prunes** it, removing the tokens whose absence hurts the least. **SentencePiece** (Kudo and Richardson, 2018) is a popular implementation that treats text as a raw stream of characters - it even encodes the space as an ordinary symbol (`▁`)[^sentencepiece]. This makes it **language-independent**, which is crucial for languages without spaces between words, like Japanese or Chinese. In practice every large model has its own tokenizer with a specific vocabulary size. For the OpenAI family (the `tiktoken` library)[^tiktoken-vocab]: | Tokenizer | Vocabulary size | Example models | |---|---|---| | `gpt2` / `r50k_base` | 50,257 | GPT-2, early GPT-3 | | `p50k_base` | 50,281 | Codex | | `cl100k_base` | ~100,277 | GPT-3.5-turbo, GPT-4 | | `o200k_base` | ~200,019 | GPT-4o, o1 | A practical rule of thumb from the `tiktoken` docs: one token is on average **about 4 characters** of English text[^tiktoken-readme]. Here is a catch that matters to us: most vocabularies are optimized mainly for English, so Polish text - with all its inflection and diacritics - breaks into **more** tokens than English of the same content. This is a real reason why the same content in Polish tends to be costlier and slower to process than in English. *(This is a general observation, not a measured number from a specific paper.)*
πŸ“ For the curious: BPE step by step Take a tiny corpus and, for a moment, write words as sequences of characters, where `Β·` marks the end of a word: ``` l o w Β· (Γ—5) l o w e r Β· (Γ—2) n e w e s t Β· (Γ—6) w i d e s t Β· (Γ—3) ``` We count the frequency of adjacent pairs. The pair `e s` appears 6 + 3 = 9 times - the most. We merge it into `es`: ``` n e w es t Β· (Γ—6) w i d es t Β· (Γ—3) ``` Now the most frequent pair is `es t` (9 times) - we merge it into `est`. Then `est Β·`, and so on. Each merge is one entry on the merge list. To tokenize a new word, we apply those merges in the same order they were created. The crux is that **frequent** sequences (like the suffix `-est`) get promoted to single tokens, while rare ones stay decomposed into finer, still-known pieces. Hence the compromise: short sequences for common words, zero OOV for rare ones.
## πŸ—ΊοΈ Embeddings: from a number to meaning We have tokens. But a token is still just a label - a row number in the vocabulary. The number alone means nothing: that "cat" has index 4711 and "dog" 4712 says nothing about them being close in meaning. We need to turn each token into a **vector** - a sequence of numbers carrying information about meaning. That is the **embedding** (literally "embedding" a word into a space of numbers). The breakthrough was **word2vec** (Mikolov et al., 2013). The idea: learn word vectors so as to predict which words appear **in the neighborhood**. Two variants[^mikolov-arch]: **CBOW** (from the context, guess the missing middle word) and **Skip-gram** (from one word, guess its context; better for rare words). The team's second paper added an efficient training method and - what fired imaginations - showed **vector analogies**[^mikolov-neurips]. It turned out that in the learned space $\text{vec}(\text{"King"}) - \text{vec}(\text{"Man"}) + \text{vec}(\text{"Woman"})$ has as its nearest neighbor $\text{vec}(\text{"Queen"})$. The direction "maleness β†’ femaleness" became an almost constant offset in the space. This is exactly that famous geometry of meaning that the interactive page shows. **But honesty is needed here** - and this is the point where popular stories about LLMs often overreach. The classic example tends to be **over-interpreted**. First, the standard method of finding the answer **excludes by design** the input words themselves ("king," "man," "woman") - without that exclusion, the nearest neighbor would often just be one of them. Linzen (2016) showed that simple baselines can be surprisingly strong, and that the method **confuses "offset consistency" with the plain structure of the neighborhood**[^linzen]. Drozd, Gladkova and Matsuoka (2016) added that the linear offset is sensitive to the idiosyncrasies of individual words and works unevenly across different types of relations[^drozd]. The conclusion worth remembering: the analogy "king - man + woman β‰ˆ queen" is a good **intuition** that the space has a linear structure - but **not a proof** that embeddings "understand" relations. It is partly an artifact of the metric and of carefully chosen examples. The interactive page shows this example uncritically (because it is an excellent visualization of the intuition) - here it is worth adding this caveat in the margin.
πŸ“ For the curious: the vector offset and the cosine metric The analogy "a is to b as c is to ?" is solved by the **vector offset** method. We look for the word $d$ whose vector is closest to the combination $\text{vec}(b) - \text{vec}(a) + \text{vec}(c)$ in terms of **cosine** similarity (cos - the cosine of the angle between vectors): $$ d^{*} = \arg\max_{w}\ \cos\!\big(\text{vec}(w),\ \text{vec}(b) - \text{vec}(a) + \text{vec}(c)\big) $$ The symbol $\arg\max$ (read "argument of the maximum") means: "the word $w$ that yields the largest value." Cosine measures **direction**, ignoring the lengths of the vectors - two vectors pointing the same way have a cosine close to 1. The key catch from the main text: $\arg\max$ traditionally **omits** the input words $a, b, c$ themselves from the search. This looks like a technical detail, but it is exactly what makes "queen" win - without that exclusion, the winner would often be one of the input words. That is why critics say the method measures the structure of the local neighborhood rather than any real "understanding" of relations.
GloVe (Pennington, Socher, Manning, 2014) took a different road[^glove]. Instead of sliding a local window over the text, GloVe learns directly from **global** statistics: how often word $i$ co-occurs with word $j$ across the entire corpus. The effect is similar - a space with interpretable geometry - but the route is different.
πŸ“ For the curious: the GloVe objective GloVe builds a co-occurrence matrix $X$, where $X_{ij}$ is the number of times word $j$ appeared in the context of word $i$. It then fits the vectors so that their dot product approximates the logarithm of that count. The minimized objective is a weighted sum of squared errors: $$ J=\sum_{i,j} f(X_{ij})\big(w_i^{\top} \tilde{w}_j + b_i + \tilde{b}_j - \log X_{ij}\big)^2 $$ The symbol $\top$ (transpose) turns a column vector into a row, so $w_i^{\top}\tilde{w}_j$ is simply the dot product of two vectors. The tilde over $\tilde{w}_j$ marks a **second**, separate set of vectors (for words appearing as context). The weighting function $f$ damps the influence of very frequent pairs (like "the"), so they do not dominate the training. The sum $\sum$ (sigma) runs over all word pairs.
### Static vs contextual - and what is *really* in an LLM word2vec and GloVe have one fundamental limitation: they are **static**. One word = one vector, regardless of the sentence. The word "bat" gets the same vector in "vampire bat" and "baseball bat" - even though these are two different things. The breakthrough came with **contextual** representations: **ELMo**[^elmo] and then **BERT**[^bert] generate a word's vector that **depends on the whole sentence**. "Bat" gets a different vector depending on its neighbors. This very ability is one of the foundations of today's LLMs. And here is a key correction, because it is an easy point to misunderstand: **an LLM does not load ready-made word2vec vectors**. The embedding layer is simply a **matrix of parameters**, trained together with the rest of the model - exactly like the weights from the previous post. Each of the $V$ tokens in the vocabulary has its row in the matrix $E$, with as many columns as the model dimension $d_{\text{model}}$ (read "d-model"). These vectors start out random and are polished by gradients during training - the network invents for itself whatever geometry of meaning is most convenient for predicting the next token. How large are these vectors in real models? The base Transformer from 2017 had $d_{\text{model}} = 512$[^vaswani]. GPT-3 (175 billion parameters) - already $d_{\text{model}} = 12288$[^gpt3]. LLaMA scaled from 4096 (the 7B variant) to 8192 (the 65B variant)[^llama]. Every token is thus represented by thousands of numbers.
πŸ“ For the curious: the embedding matrix and weight tying The embedding matrix is $E \in \mathbb{R}^{V \times d_{\text{model}}}$ - $V$ rows (one per vocabulary token) and $d_{\text{model}}$ columns. A "token's embedding" is literally pulling out the corresponding row. With a vocabulary of $V \approx 50{,}000$ and $d_{\text{model}} = 12288$, this matrix alone has hundreds of millions of parameters. **Weight tying.** Press and Wolf (2017) noticed that the matrix the model uses at the **output** (to turn vectors back into token probabilities) is itself a valid word embedding[^press-wolf]. They proposed **sharing** a single matrix at the input ($E$) and at the output (as $E^{\top}$, that is, the transpose). This saves hundreds of millions of parameters and usually improves quality (a lower perplexity - the "surprise" from the previous post). Hence its prevalence in GPT-2.
## 🧭 How the model knows in what order the words came There is one more problem that primers usually pass over, and it is absolutely crucial. The attention mechanism - the heart of the Transformer, which we reach in the next post - has a surprising property: it is **order-insensitive**. By itself it does not tell "the dog bit the postman" apart from "the postman bit the dog." It looks at a set of tokens, not at their arrangement. And yet word order carries meaning. So information about **position** has to be injected separately. Several ideas emerged for how to do this. **Sinusoidal encoding (absolute).** The original Transformer added to each embedding a fixed vector built from sines and cosines of different frequencies[^vaswani] - something like a unique "fingerprint" of the position. Each position in the sentence gets its characteristic pattern, and the model learns to read it. **Learned encoding.** A simpler alternative: a separate trained table of vectors, one per position (this is what BERT and GPT-2 do). The quality comes out almost identical to the sinusoids[^vaswani], but there is a drawback: the model does not naturally handle sequences **longer** than those it saw in training - for a position it never trained on, it simply has no vector. **RoPE (Rotary Position Embedding).** Today's favorite (Su et al., 2021)[^rope], used among others in LLaMA. Instead of **adding** position, RoPE **rotates** the query and key vectors by an angle proportional to position. The idea is elegant: after the rotation, the dot product of two vectors depends only on the **difference** of their positions, not on the absolute positions. The model thus encodes position **relatively** ("how much further along"), which better reflects how language actually works. **ALiBi (Attention with Linear Biases).** Another clever idea (Press, Smith, Lewis, 2021)[^alibi]: do not touch the embeddings at all, just add to the attention scores a **linear penalty** that grows with the distance between tokens. The farther apart, the stronger the damping. The main advantage: a model trained on short sequences can **extrapolate** to much longer ones at inference time.
πŸ“ For the curious: the sinusoidal formula and the intuition behind RoPE Sinusoidal encoding for position $pos$ and dimension $i$: $$ PE_{(pos,\,2i)}=\sin\!\Big(\frac{pos}{10000^{2i/d_{\text{model}}}}\Big),\qquad PE_{(pos,\,2i+1)}=\cos\!\Big(\frac{pos}{10000^{2i/d_{\text{model}}}}\Big) $$ Even dimensions get a sine, odd ones a cosine. Different dimensions have different frequencies (through the divisor $10000^{2i/d_{\text{model}}}$) - low dimensions "spin" fast, high ones slowly. It is like a clock with many hands: the combination of their positions uniquely encodes the position, and neighboring positions have similar patterns. RoPE does this differently: it splits the vector into pairs of coordinates and **rotates** each pair by an angle that depends on position (like a hand on a dial). Because the dot product of two rotated vectors depends only on the difference of angles - that is, on the **difference of positions** $m-n$ - the model gets relative-distance information for free, without adding anything to the embedding.
## πŸŒ— A geometry no one designed Let us pause for a moment on what we have just described - because in the technical density it is easy to miss something astonishing. We started from **discrete symbols**: a token is just a number in a vocabulary, a label with no content. And we ended at a **continuous space** in which directions correspond to concepts, closeness means similarity, and relations (at least roughly) fall into geometric offsets. No one designed this geometry by hand. It emerged on its own - as a side effect of the only pressure we know from the previous post: "predict the next token as well as you can." That is why I kept the honest critique of the famous vector analogies in this post - not to strip the geometry of its wonder, but so as not to ascribe to it *more* than the data show: "king - man + woman β‰ˆ queen" holds less often and less cleanly than the legend claims. And even so, something striking remains without any exaggeration - that **discrete** symbols, labels with no content, can be embedded at all in a space where distance begins to *resemble* similarity, and direction - relation. And here comes the boldest hypothesis of the whole series, and at the same time the one that fits this post best - because it bears directly on the **geometry of representation**. It claims that the space described above is not a chance artifact of a single model: that networks of different builds, trained on different data, and perhaps different **substrates** (the brain among them), converge toward **the same** structure. It has been named the Platonic Representation Hypothesis; I develop it - with due caution - in a [separate essay on the human-LLM resonance](/en/posts/rezonans-czlowiek-llm-kwantowe-lustro). ## 🎯 What's next We have built the bridge between text and numbers. Text breaks into **tokens** (subword - not words, not letters), and each token gets a **vector** from a trained embedding matrix - a vector that over time takes on a geometry of meaning (with due skepticism toward the famous "king - man + woman" analogy). We also added the encoding of **position**, without which the model would be blind to word order. So we have a sequence of vectors, each aware of its place in the sentence. But the vectors still sit side by side in isolation - they do not "talk" to one another. And yet meaning depends on neighbors: "bank" means something different by a river than in finance, and in "the knife didn't cut the bread because it was too blunt" the word "it" points to the knife - but change "blunt" to "stale" and it points to the bread instead. In the next stop of the series we reach the heart of the Transformer - the **attention mechanism**: how each token "looks around" at the others and takes from them what it needs to sharpen its own meaning. If you would rather **touch** now than read on, the [interactive page "Anatomy of an LLM"](/en/llm/anatomy#tokeny) lets you see how text splits into tokens and how the vectors arrange themselves in space. And if what tempts you is what really *emerges* from the geometry described here - and why different minds seem to converge toward a shared structure of meaning - [I wrote about that separately](/en/posts/rezonans-czlowiek-llm-kwantowe-lustro). [^gage]: P. Gage (1994). *A New Algorithm for Data Compression*. The C Users Journal (BPE as a compression algorithm). [^sennrich]: R. Sennrich, B. Haddow, A. Birch (2016). *Neural Machine Translation of Rare Words with Subword Units*. ACL. arXiv:1508.07909. [^gpt2]: A. Radford et al. (2019). *Language Models are Unsupervised Multitask Learners* (GPT-2; byte-level BPE), OpenAI. [^schuster]: M. Schuster, K. Nakajima (2012). *Japanese and Korean Voice Search* (WordPiece). ICASSP. [^gnmt]: Y. Wu et al. (2016). *Google's Neural Machine Translation System* (GNMT; ~32,000 word-pieces). arXiv:1609.08144. [^bert]: J. Devlin et al. (2019). *BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding*. NAACL. arXiv:1810.04805. [^kudo-unigram]: T. Kudo (2018). *Subword Regularization: Improving NMT Models with Multiple Subword Candidates* (the Unigram model). ACL. arXiv:1804.10959. [^sentencepiece]: T. Kudo, J. Richardson (2018). *SentencePiece: A simple and language independent subword tokenizer*. EMNLP. arXiv:1808.06226. [^tiktoken-vocab]: OpenAI, `tiktoken` - vocabulary sizes from `tiktoken_ext/openai_public.py`. [^tiktoken-readme]: OpenAI, `tiktoken` - README (the "each token corresponds to about 4 bytes" rule). [^mikolov-arch]: T. Mikolov et al. (2013). *Efficient Estimation of Word Representations in Vector Space* (CBOW, Skip-gram). arXiv:1301.3781. [^mikolov-neurips]: T. Mikolov et al. (2013). *Distributed Representations of Words and Phrases and their Compositionality* (negative sampling, vector analogies). NeurIPS. arXiv:1310.4546. [^linzen]: T. Linzen (2016). *Issues in evaluating semantic spaces using word analogies*. RepEval. arXiv:1606.07736. [^drozd]: A. Drozd, A. Gladkova, S. Matsuoka (2016). *Word Embeddings, Analogies, and Machine Learning: Beyond king - man + woman = queen*. COLING. [^glove]: J. Pennington, R. Socher, C. D. Manning (2014). *GloVe: Global Vectors for Word Representation*. EMNLP. [^elmo]: M. E. Peters et al. (2018). *Deep contextualized word representations* (ELMo). NAACL. arXiv:1802.05365. [^vaswani]: A. Vaswani et al. (2017). *Attention Is All You Need* ($d_{\text{model}}$, sinusoidal and learned encoding). arXiv:1706.03762. [^gpt3]: T. B. Brown et al. (2020). *Language Models are Few-Shot Learners* (GPT-3; $d_{\text{model}}=12288$, Tab. 2.1). arXiv:2005.14165. [^llama]: H. Touvron et al. (2023). *LLaMA: Open and Efficient Foundation Language Models* (dimensions 4096-8192; RoPE). arXiv:2302.13971. [^press-wolf]: O. Press, L. Wolf (2017). *Using the Output Embedding to Improve Language Models* (weight tying). EACL. arXiv:1608.05859. [^rope]: J. Su et al. (2021). *RoFormer: Enhanced Transformer with Rotary Position Embedding* (RoPE). arXiv:2104.09864. [^alibi]: O. Press, N. A. Smith, M. Lewis (2022). *Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation* (ALiBi). ICLR. arXiv:2108.12409.