--- title: "Anatomy of an LLM: from a single neuron to the attention mechanism" date: 2026-06-08 description: "How is a language model really built? A guide without magic: neuron, weights, tokens, embeddings, the attention mechanism, the Transformer block and LoRA - step by step, with interactive diagrams to play with yourself." tags: ["llm anatomy", "ai", "llm", "neural networks", "transformer", "deep-dive"] --- ## πŸš€ Intro You type a single sentence to ChatGPT, and a second later back comes a paragraph that reads as if written by a human. You ask for a poem - you get a poem. For code - you get code. For a solution to a problem you spent an hour stuck on yourself - and the model breaks it down into pieces, as if it genuinely *understood* what you are asking. Underneath, though, there is no one sitting there and thinking. There is math: text turned into numbers, and those numbers pushed through a gigantic series of matrix multiplications. All of the model's "knowledge" is billions of weights - numbers polished during training. And it is exactly those building blocks that we will take apart in this post: the neuron, the weights, the tokens, the attention mechanism. Except the story does not end there - and this is where it gets most interesting. Because out of those simple multiplications **something emerges that no one programmed in directly**. The engineers only wanted the model to guess the next word more accurately. What they got was a system that can reason, follow a train of thought, operate something like an internal imagination - and one that, the better it gets, the more its internal representations [begin to structurally resemble the human brain](/en/posts/rezonans-czlowiek-llm-kwantowe-lustro), even though no one built it that way. The truth is that **we do not fully know why matrix multiplication acquires that "something"**. And there is a real chance that the structure emerging here is deeper than we currently suspect. That is why I will not treat this as stripping the magic away. Quite the opposite: to appreciate how astonishing what language models do really is, you first have to see what they are built from. Understanding the mechanism does not take the mystery away - it shows you exactly where it lives. This post is a map of that construction: **from a single artificial neuron all the way to the attention mechanism and cheap fine-tuning with LoRA**. We will walk it in seven stops, without requiring math beyond high-school level. At each stop you will find a link to the [interactive page](/en/llm/anatomy), where you can touch the same piece: drag the sliders, watch the chart, hover over a neuron. > πŸ’‘ **How to read this text.** The main thread is descriptive - just reading it is enough. I tuck the formulas into collapsible "for the curious" blocks: if you like math, expand them; if not, skip them with no loss to understanding. ## πŸ“‹ TL;DR - **A neural network is millions of copies of one simple element** - the artificial neuron, which multiplies numbers by weights, sums them up and passes the result through an activation function. - **The activation function is the source of "nonlinearity"** - without it the whole network, even a thousand layers deep, would be equivalent to a single multiplication and would not learn anything complex. - **A model's "parameters" (those famous 7B, 70B) are simply all of its weights.** Training is the tedious work of correcting these numbers billions of times until the model's predictions become accurate. - **Text turns into numbers** through tokens (chunks of words) and embeddings (vectors of meaning), in which geometry carries sense: words with similar meaning lie close to one another. - **The attention mechanism is the heart of the Transformer** - it lets every word "look around" the whole sentence and decide which other words matter to it. This is what made today's LLMs possible. - **An LLM is a tall stack of identical Transformer blocks**, and at the very end - drawing the next token at random from a probability distribution. - **LoRA is a clever way to fine-tune cheaply**: we freeze the huge model and train only a tiny "add-on", saving hundreds of times the resources. - You can **touch and drag** everything below on the [interactive page](/en/llm/anatomy). ## 🧱 The building block of everything: a single neuron Let's start with the smallest part. The whole neural network, no matter how gigantic, is made of millions of copies of one genuinely simple element. The **artificial neuron** does just one thing: it takes a few numbers on its input, multiplies each by an assigned **weight**, adds them all together, throws in one more number called the **bias** (an offset), and passes the result through what is called the **activation function**. The weight is the heart of the whole story. It tells the neuron *how important* a given input is: a large positive weight amplifies the signal, a negative one suppresses it, one close to zero practically ignores it. When we say a model "learned something", we mean exactly that these weights settled into the right values. The idea is not new. The first formal model of a neuron was proposed in 1943 by the neurophysiologist Warren McCulloch and the logician Walter Pitts - it was a simple threshold unit that "fired" when the sum of its inputs crossed a threshold. The modern neuron is a softened, "smooth" version of that idea.
πŸ“ For the curious: the neuron equation First we compute the weighted sum $z$, then pass it through the function $f$: $$ z = \sum_{i=1}^{n} w_i x_i + b \qquad\Longrightarrow\qquad y = f(z) = f\!\left(\mathbf{w}\cdot\mathbf{x} + b\right) $$ The notation $\mathbf{w}\cdot\mathbf{x}$ is the *dot product* - shorthand for writing the sum $w_1x_1 + w_2x_2 + \dots$. That is all: one multiplication, one addition, one function.
πŸ‘‰ **Play with it:** on the interactive page there is a [neuron lab](/en/llm/anatomy#neuron) - drag the weight and input sliders, switch the activation function and watch the result change live. ## 〰️ Nonlinearity: why the activation function is necessary Here comes a subtlety that decides everything. If a neuron only summed and multiplied, then stacking even a thousand layers could be folded back down into a single multiplication. The whole network would then be equivalent to a straight line - it could model only linear relationships, that is, almost nothing we care about (language, images, logic are not linear). The rescue is the **activation function**: a small "bend" inserted at the output of every neuron. It is what introduces **nonlinearity**, thanks to which the network can model arbitrarily wavy, complicated relationships. Mathematicians even proved that a network with such a nonlinearity can in principle approximate *any* continuous function - this is the so-called universal approximation theorem. There are several families of activation functions. The classic and cheap one is **ReLU** (simply "zero out negative numbers"). The older ones are **sigmoid** and **tanh**, which squeeze the result into a narrow range. Modern LLMs use "smooth" variants such as **GELU** or **SwiGLU** - they are continuous and have a well-defined derivative everywhere, which makes learning much easier. πŸ‘‰ **Play with it:** the [interactive activation-function chart](/en/llm/anatomy#aktywacja) lets you switch between ReLU, Sigmoid, tanh and GELU and see how their shapes really differ. ## πŸ•ΈοΈ From neuron to network - and where the "billions of parameters" come from A single neuron is weak. The power appears when we line up hundreds of them side by side into a **layer**, and stack layers one behind another. The outputs of one layer become the inputs of the next, and the signal flows from input to output - this is called *forward propagation*. Here lies the reason LLMs are trained on graphics cards. A whole layer can be computed with a single **matrix multiplication**, and GPUs are designed precisely for multiplying huge tables of numbers at lightning speed. All of the model's "knowledge" sits in the numbers inside these matrices. And this is where the riddle of the mysterious "7B" or "70B" gets solved. When you hear that a model has 7 billion parameters, that means exactly the **number of all the weights and biases** in the network. A single layer connecting a thousand inputs to a thousand outputs already has over a million weights - and there are hundreds of such layers in a model. Hence the billions. No one sets these numbers by hand. The model starts from random values and corrects them in a loop, repeated billions of times: **(1)** it guesses the next word, **(2)** checks how badly it was wrong (the "error", or *loss*), **(3)** computes which way to nudge each weight to shrink the error (the gradient, computed by *backpropagation*), **(4)** takes a tiny step in that direction. Repeat enough times, and out of random noise emerges a model that "understands" language.
πŸ“ For the curious: a layer and a training step We write a whole layer with a single formula, where $W$ is the weight matrix and $\mathbf b$ is the bias vector: $$ \mathbf{h} = f\!\left(W\mathbf{x} + \mathbf{b}\right) $$ A single weight is corrected by the gradient descent rule: $$ w \leftarrow w - \eta \cdot \frac{\partial L}{\partial w} $$ where $L$ is the error and $\eta$ (eta) is the *learning rate* - how big a step we take each time.
How much text, and how big a model? It turns out this is not arbitrary - there are empirical *scaling laws*, and the popular rule of thumb (known as Chinchilla) says that for every model parameter it is worth allocating roughly 20 tokens of training data. πŸ‘‰ **Play with it:** see the [diagram of a fully connected network](/en/llm/anatomy#siec) (hover over a neuron to highlight its connections) and the [training loop](/en/llm/anatomy#parametry). ## πŸ”€ How text becomes numbers: tokens and embeddings The network does not understand letters - it understands only numbers. So before a sentence reaches the model, something clever happens to it. First the text is cut into **tokens**, that is chunks of words (sometimes a whole word, sometimes a fragment - e.g. "un", "believ", "able"). Then each token is turned into an **embedding** - a long vector of numbers, something like the "coordinates" of a word in a multidimensional space of meanings. The most beautiful part is that in this space **geometry carries meaning**. Words with similar sense land close together - "cat" near "dog". What is more, the relations between words themselves become directions in this space. The classic (though nowadays somewhat criticized as a simplification) example: if you take the vector of the word "king", subtract "man" and add "woman", you land roughly in the neighborhood of the word "queen".
πŸ“ For the curious: the arithmetic of meanings $$ \vec{\text{king}} - \vec{\text{man}} + \vec{\text{woman}} \;\approx\; \vec{\text{queen}} $$ The fact that such "arithmetic on words" works at all was one of the discoveries that ignited interest in embeddings.
πŸ‘‰ **Play with it:** [see how a sentence breaks down into tokens and vectors](/en/llm/anatomy#tokeny). ## πŸ‘οΈ The attention mechanism - the heart of the Transformer This is the breakthrough that makes us talk about LLMs at all today. **Attention** lets every word "look around" the whole sentence and decide for itself which other words matter most to it. Take the sentence: "The cat didn't eat the fish, because it *was* spoiled." What does "it" refer to? A human knows instantly that it is the fish, not the cat. The attention mechanism lets the model do exactly the same - connect "it" with "fish", even though several words separate them. It works through three roles that the model creates for each token: - **Query** - "what am I looking for?" - **Key** - "what do I offer?", that is the label of each token. - **Value** - "what content I will pass on if I get chosen". The model compares one word's query against the keys of all the others, from that comparison it computes the **attention weights** (what percentage of attention to devote to each word), and finally it blends the values in proportion to those weights. Importantly, the model does this on **dozens of "heads" at once** (multi-head): one head can track grammar, another references, a third style - and their results get combined.
πŸ“ For the curious: scaled dot-product attention $$ \text{Attention}(Q,K,V) = \text{softmax}\!\left(\frac{Q K^{\top}}{\sqrt{d_k}}\right) V $$ The product $QK^{\top}$ is the matching of queries to keys. The upper "$\top$" next to $K$ denotes *transposition* - rotating the matrix (swapping rows with columns), a technical step needed so that every query can be multiplied by every key. The lower "$k$" in $d_k$ is just an ordinary label: $d_k$ is the *dimension of the keys*, that is how many numbers make up a single Key vector (the same as a Query vector). Dividing by $\sqrt{d_k}$ (the square root of that dimension) stabilizes the values, so the *softmax* function does not "saturate" for large numbers. Softmax turns the raw scores into attention percentages that sum to 1.
πŸ‘‰ **Play with it:** [the attention visualization](/en/llm/anatomy#uwaga) - pick a query word and see which other words it directs its attention to. ## 🧊 The Transformer block: everything together A single attention mechanism is not yet a model. An LLM is a **tall stack of identical Transformer blocks** - sometimes 32, 80, and in the largest models over a hundred of them. Each such block combines two things: an *attention* layer (which we met above) and an ordinary *feed-forward network* (FFN), which processes the result further. To make it possible to train such deep towers, two technical tricks are thrown in: **residual connections** (the input is added to the output, giving the gradient a "highway" deep into the network) and **normalization** (keeping the numbers in a healthy range). They sound unremarkable, but without them very deep networks simply refuse to learn. At the very top of the stack, the last layer turns the vectors back into **probabilities of the next token**. And here is the punchline that surprises a lot of people: what we perceive as a finished, thought-through utterance is, at the level of the mechanism, a probability distribution over all possible next tokens, from which the model *draws* one at random. Then it glues it onto the text and repeats the whole thing from the start, token by token. The entire fluent answer is simply "guess the next word" repeated many times over. πŸ‘‰ **Play with it:** [the diagram of a single Transformer block](/en/llm/anatomy#transformer) shows how attention, the FFN and the residual connections come together into one whole. ## πŸͺΆ LoRA: cheap fine-tuning of huge models The last stop is the answer to a very practical problem. Suppose you have a ready 70B model and you want to specialize it - for example teach it a legal style or customer support. Full fine-tuning would mean updating *all* 70 billion parameters. That requires enormous graphics cards and serious money. **LoRA** (Low-Rank Adaptation) is an elegant shortcut. Instead of touching the original weights, we **freeze** them entirely and attach a tiny "add-on" on the side - a pair of thin matrices that learn the needed change. The key observation is that the change needed to specialize a model is "low-rank", meaning it can be compressed into two skinny matrices instead of one large one. The result? For typical settings LoRA trains as many as **several hundred times fewer parameters** than full fine-tuning. The resulting "adapter" weighs megabytes instead of gigabytes, so you can keep many adapters (one "legal style", another "customer support") and plug them into the same base model on demand - a bit like interchangeable lenses for a single camera.
πŸ“ For the curious: what "low-rank" means Instead of learning a full correction matrix $\Delta W$ of size $d\times d$, we learn two thin matrices $A$ and $B$: $$ W' = W_0 + \Delta W = W_0 + \frac{\alpha}{r}\,B A,\qquad B\in\mathbb{R}^{d\times r},\; A\in\mathbb{R}^{r\times d},\; r \ll d $$ The number $r$ is the *rank* (e.g. 8 or 16) - the bottleneck that decides the number of new parameters; $\alpha$ is the scale of the add-on's influence, and $W_0$ stays untouched. For $d = 4096$ and $r = 8$ this is the difference between ~16.8M and ~65k parameters, that is about **256x fewer**.
πŸ‘‰ **Play with it:** [the LoRA diagram](/en/llm/anatomy#lora) shows how two thin matrices are attached to the frozen matrix, forming a cheap "side track". ## 🎯 What's next That was the whole journey: from a single number multiplied by a weight, through nonlinearity, layers and training, tokens and embeddings, all the way to the attention mechanism, the Transformer block and LoRA. We assembled the model from parts - and that is exactly why we can now honestly say where what we understand ends. We know *how* the mechanism is built. We still do not know *why* the ability to reason, follow a train of thought and operate an internal imagination emerges from those multiplications. The anatomy did not solve the mystery - it only showed exactly where it sits. If you now want to **touch** these concepts instead of just reading about them, the [interactive "Anatomy of an LLM" page](/en/llm/anatomy) walks through these same eight sections with sliders, charts and diagrams you can experiment with live. From here two roads lead onward. **Deeper into the mechanism** - each of these stops has its own fuller, cited story (its origin, the exact formulas, numbers from real models), which appear as further posts in this series; all six are now up: [the neuron and activation functions](/en/posts/llm-neuron-i-aktywacje) (from McCulloch and Pitts' threshold to SwiGLU), [networks, parameters and training](/en/posts/llm-sieci-parametry-trening) (from matrix multiplication to Chinchilla's scaling laws), [tokens and embeddings](/en/posts/llm-tokeny-embeddingi) (from subword tokenization to the geometry of meaning and positional encoding), [the attention mechanism](/en/posts/llm-mechanizm-uwagi) (from the bottleneck in translation networks to FlashAttention and induction heads), [the Transformer block](/en/posts/llm-blok-transformera) (from the three architecture families, through residual connections and normalization, to Mixture of Experts) and [LoRA and fine-tuning](/en/posts/llm-lora-dotrenowywanie) (from the low-rank hypothesis, through QLoRA, to instruction tuning and RLHF). And **toward the mystery** - if you are curious about what actually *emerges* from this machinery and why a conversation with a model can resemble a resonance with another mind, [I wrote about that separately](/en/posts/rezonans-czlowiek-llm-kwantowe-lustro). ## πŸ“š Sources and further reading The hub deliberately stays at the level of intuition. Below are the most important source works for those who want to reach for the originals; fuller, cited bibliographies will appear in the deep-dive posts of the series. **Classic works - the foundations** - W. S. McCulloch, W. Pitts, *"A Logical Calculus of the Ideas Immanent in Nervous Activity"*, Bulletin of Mathematical Biophysics, vol. 5, 1943, pp. 115-133 - the first formal model of an artificial neuron. - G. Cybenko, *"Approximation by Superpositions of a Sigmoidal Function"*, Mathematics of Control, Signals and Systems, vol. 2, 1989, pp. 303-314 - the classic universal approximation result (generalized by Hornik, 1991). **Architecture and training of modern LLMs** - A. Vaswani et al., *"Attention Is All You Need"*, NeurIPS 2017. arXiv: https://arxiv.org/abs/1706.03762 - the paper that introduced the Transformer architecture and the attention mechanism. - J. Hoffmann et al., *"Training Compute-Optimal Large Language Models"* (Chinchilla), 2022. arXiv: https://arxiv.org/abs/2203.15556 - the source of the "~20 tokens per parameter" rule. - E. J. Hu et al., *"LoRA: Low-Rank Adaptation of Large Language Models"*, 2021. arXiv: https://arxiv.org/abs/2106.09685 - the cheap fine-tuning method described in the last section.