--- title: "Anatomy of an LLM: the neuron and activation functions" date: 2026-06-09 description: "The smallest building block of a language model under the microscope: where the artificial neuron came from, why without an activation function the whole network would be useless, and how ReLU, GELU and SwiGLU really differ - with formulas, history and concrete details from GPT, LLaMA and Gemma." tags: ["llm anatomy", "ai", "llm", "neural networks", "deep-dive"] --- ## 🚀 Intro In [the overview post of this series](/en/posts/anatomia-llm) we took a language model apart into seven stops - from the neuron to LoRA - in a way that let you walk the whole thing without a single formula. That was the bird's-eye view. Now we come down to earth and begin the most serious part of the journey: **into the mechanism**, stop by stop, this time with the history, full formulas and concrete numbers from real models. We start at the very bottom - at the building block out of which everything else is made: the single artificial neuron and that small "bend" on its output we call the activation function. It looks like the most boring element of the whole puzzle. And yet: without that bend even the largest model would collapse into a single multiplication and would learn nothing interesting, and the quiet evolution of these functions - from the sigmoid to SwiGLU - turned out to be one of the heroes behind the leap in quality of modern LLMs. There is one more thing worth keeping in mind right at the entrance. The artificial neuron we are about to take apart is a **drastic caricature** of its biological original. And even so, out of billions of such caricatures something emerges that their formula in no way foretells. That tension - the simplicity of the parts, the unobviousness of the whole - will accompany us through the entire series. > 💡 **How to read this text.** The main thread is descriptive and self-contained. I tuck the formulas into collapsible "for the curious" blocks - if you like math, expand them; if not, skip them with no loss to understanding. Every claim has a footnote to its source paper at the end. ## 📋 TL;DR - **The artificial neuron** computes a weighted sum of its inputs, adds a bias and passes the result through an activation function. The idea is 80 years old (McCulloch and Pitts, 1943) and is a deliberate, heavy simplification of biology. - **The activation function is the only source of nonlinearity.** Without it, a stack of even a thousand layers is mathematically equal to a single layer - that is, to an ordinary straight line. - **The universal approximation theorem** says a network with a single hidden layer can approximate any continuous function. But it is a theorem about *existence* - it promises neither that such a network will be small nor that gradient descent will find it. - **ReLU** (zero out negative numbers) displaced the sigmoid because it does not "saturate" and does not kill the gradient in deep networks. Its weakness - "dead" neurons - was patched by variants. - **Modern LLMs use smooth, gated activations:** GPT - GELU, LLaMA and PaLM - SwiGLU, Gemma - GeGLU. This is not cosmetics: in benchmarks they gave the best quality. - Everything below you can **touch** on the [interactive page](/en/llm/anatomy#neuron). ## 🧠 The neuron model: from McCulloch and Pitts' threshold to today The history of the artificial neuron begins with a question that sounds audacious today: can thinking be described by logic? **McCulloch and Pitts (1943).** The first formal model of an artificial neuron is a binary **threshold unit**. It sums its inputs and "fires" (output 1) if the sum exceeds a fixed threshold; otherwise it stays silent (output 0). The neurophysiologist Warren McCulloch and the logician Walter Pitts showed that networks of such units can realize Boolean logic, and with the right connectivity are computationally universal[^mcculloch]. From that pair - a physician and a logician - came an idea that drives the entire industry today. **Rosenblatt's perceptron (1958).** Frank Rosenblatt went a step further and added what was missing: **learning**. His perceptron is a single-layer network of units with weighted inputs and a threshold activation, with an explicit rule for tuning the weights on examples[^rosenblatt]. It is the first practically realized model of a neural network - and the first that "learned" from data instead of being programmed by hand. **Winter: the XOR limitation (1969).** The enthusiasm was quickly cooled. In the book *Perceptrons*, Marvin Minsky and Seymour Papert proved that a single-layer perceptron represents only **linearly separable** functions and cannot compute even an operation as simple as XOR[^minsky]. The result - often over-interpreted - contributed to the "winter" of neural-network research in the 1970s. The way out came only with **multi-layer** networks trained by backpropagation, which got around that limitation. (We will return to layers and training in the next post of the series.) **The modern neuron** is a "softened", smooth version of that threshold unit: instead of a hard "fire / stay silent" we have a continuous activation function. The rest is exactly the same idea - a weighted sum plus a bias.
📐 For the curious: the neuron equation The neuron computes a weighted sum of inputs $x_i$ with weights $w_i$, adds a bias $b$, and passes the result $z$ through an activation function $\phi$ (phi): $$ y = \phi\!\left(\sum_{i} w_i x_i + b\right) = \phi(\mathbf{w}\cdot\mathbf{x} + b) $$ The notation $\mathbf{w}\cdot\mathbf{x}$ is the dot product - shorthand for the sum $w_1x_1 + w_2x_2 + \dots$. In the McCulloch-Pitts model $\phi$ was a step (a threshold); today it is a smooth function, which we will meet in a moment.
Before we go further, one honest caveat. The word "neuron" suggests we are reproducing the brain. We are not. The artificial neuron is a **drastic idealization**[^minsky], and it is worth knowing exactly what was thrown overboard: - **No spikes.** Biological neurons communicate by voltage spikes (action potentials) - "all-or-nothing" events that encode information also in *time* and in the pattern of firing. The artificial neuron has a single continuous number, interpreted at most as an averaged rate. - **No memory.** A real neuron is a dynamical system - its state depends on its history. A standard feedforward neuron is **memoryless**: its output depends only on the current input. Time is added later by the architecture (e.g. by autoregression in an LLM), not by the neuron itself. - **Linear summation.** Real dendrites perform complex, nonlinear, location-dependent integration; here we have a simple weighted sum. - **Rigid synapses.** The weights are fixed during a single pass; biological synapses have rich, time-dependent plasticity. This is not a footnote for pedants. It is the first time in this series that the gulf is visible between the simplicity of the building block and the complexity of what grows out of it - and the reason why humility about the "thinking" of models is in order. ## 〰️ Why nonlinearity is necessary Here comes a fact that decides everything - and that is easy to overlook. If a neuron only multiplied and summed, then composing layers would be composing linear transformations, and **a composition of linear functions is again a linear function**. In other words: a stack of a thousand layers without activations collapses, algebraically, to a single layer. The whole gigantic network would be able to model only linear relationships - that is, almost nothing of what interests us, because language, image and logic are not linear. The rescue is precisely the activation function - a small "bend" inserted on the output of every neuron. It is what introduces **nonlinearity**, thanks to which the layers stop cancelling out and the network gains the power to model arbitrarily wavy relationships. How much power? Surprisingly much - and it was proven formally. The **universal approximation theorem** (Cybenko, 1989) says that a feedforward network with a **single** hidden layer and a continuous sigmoidal nonlinearity can approximate **any** continuous function to any accuracy[^cybenko]. Two years later Hornik generalized the result: what is decisive is the **multi-layer architecture** itself, not the particular choice of activation function[^hornik]. This sounds like a magical promise - one layer is enough for everything. Except the promise has fine print, and the kind worth understanding, because it guards against naivety: - **It is a theorem about *existence*.** It guarantees that an appropriate network *exists* - it says nothing about how large it must be or how to find it. - **The width can be exponential.** For some functions a shallow (single-layer) network would have to have a number of neurons growing exponentially with the input dimension. **Deep** networks represent the same functions far more cheaply - this is the formal advantage of depth proven by Telgarsky[^telgarsky]. That is why we build deep models, not wide ones. - **No guarantee of learnability.** The theorem says a set of weights *exists* - it does not promise that gradient descent will reach it (the loss landscape is non-convex, full of local minima). This distinction - "a solution exists" versus "we can find it" - is one of the most important motifs in all of machine learning. We will return to it when we talk about training.
📐 For the curious: what the theorem says exactly Cybenko showed that finite linear combinations of compositions of a fixed sigmoidal function $\sigma$ (sigma; with affine transformations) are **dense** in the space of continuous functions on the unit hypercube $[0,1]^n$. Formally: for any continuous $g$ and any arbitrarily small error tolerance $\varepsilon>0$ (epsilon) there exists a sum $$ G(\mathbf{x}) = \sum_{j=1}^{N} \alpha_j\,\sigma\!\left(\mathbf{w}_j\cdot\mathbf{x} + b_j\right), \qquad \text{such that}\quad |G(\mathbf{x}) - g(\mathbf{x})| < \varepsilon $$ for all $\mathbf{x}$ in the domain[^cybenko]. The assumptions matter: the target function must be **continuous**, and the domain **compact** (closed and bounded). The theorem gives no rate - how many neurons $N$ are needed for a given $\varepsilon$ - and it is exactly this gap that the depth-advantage results fill[^telgarsky].
👉 **Play with it:** on the interactive page you have a [neuron lab](/en/llm/anatomy#neuron) - drag the weights and inputs and watch the weighted sum change live, plus an [activation-function chart](/en/llm/anatomy#aktywacja) on which you can switch between the shapes discussed below. ## 🔧 The zoo of activation functions and its evolution If nonlinearity is necessary, the question remains: *which* bend should we choose? The answer evolved over decades, and each change solved a concrete, tangible problem of its predecessor. Let us trace that line. ### Sigmoid and tanh: an elegance that chokes the gradient The first popular activations were smooth and mathematically "pretty". **Sigmoid** squeezes any number into the range $(0,1)$, and **tanh** into $(-1,1)$. The problem shows up in deep networks: for large input values both functions **saturate** - their graph goes flat and the slope (the derivative) tends to zero. Why is this a catastrophe? Because learning consists of pushing the error signal backward through all the layers, multiplying along the way by these slopes. When each of them is a fraction close to zero, the product of several dozen such fractions becomes microscopic - this is the famous **vanishing gradient**. The effect: early layers learn at a snail's pace, or not at all. This was one of the main walls that deep learning kept hitting before ReLU was popularized[^hendrycks].
📐 For the curious: formulas and derivatives of sigmoid and tanh We write the sigmoid with the small Greek letter $\sigma$ (sigma), and the hyperbolic tangent as $\tanh$: $$ \sigma(x) = \frac{1}{1+e^{-x}}, \qquad \sigma'(x) = \sigma(x)\big(1-\sigma(x)\big) $$ $$ \tanh(x) = \frac{e^x - e^{-x}}{e^x + e^{-x}}, \qquad \tanh'(x) = 1 - \tanh^2(x) $$ The maximum of the sigmoid's derivative is a mere $0.25$ (at $x=0$), and tanh's - $1$ (also at $x=0$). For large $|x|$ both derivatives tend to zero: hence the gradient vanishing in the product over many layers. Tanh has an extra advantage: it is centered around zero, which usually gives healthier learning dynamics than the sigmoid with its range $(0,1)$.
### ReLU: brutally simple and therefore effective The breakthrough turned out to be surprisingly primitive. **ReLU** (Rectified Linear Unit) is simply: "zero out negative numbers, pass positive ones through unchanged". No exponentials, no saturation on the positive side - there the slope is exactly 1, so the gradient flows freely through even hundreds of layers. On top of that it is lightning-fast to compute and produces **sparse** representations (some neurons output a clean zero), which itself helps. A curiosity: ReLU did not fall from the sky as an engineering trick. It emerged from theory - Nair and Hinton derived it by summing infinitely many copies of a binary unit with shifted thresholds, which yields a smooth approximation called softplus[^nair]. A year later Glorot, Bordes and Bengio showed empirically that rectifying neurons match or beat tanh, while producing the desired sparsity[^glorot]. ReLU has, however, its own Achilles' heel: **"dying" neurons** (dying ReLU). For a negative input the derivative is 0, so if a neuron gets stuck with a negative sum for all examples, its gradient is permanently zero - the neuron stops learning and is practically dead. A whole family of patches arose for this ailment.
📐 For the curious: ReLU and its variants against "dead" neurons $$ \mathrm{ReLU}(x) = \max(0, x), \qquad \mathrm{ReLU}'(x) = \begin{cases}1 & x>0\\ 0 & x<0\end{cases} $$ (at zero the function is non-differentiable - in practice one takes the derivative as 0 or 1). The variants introduce a nonzero slope on the negative side, controlled by a parameter $\alpha$ (alpha), so the neuron never freezes completely - like **Leaky ReLU**[^maas] with a fixed small $\alpha\approx 0.01$ or **ELU**: $$ \mathrm{LeakyReLU}(x) = \begin{cases} x & x>0\\ \alpha x & x\le 0\end{cases} \qquad \mathrm{ELU}(x) = \begin{cases} x & x>0\\ \alpha(e^x - 1) & x\le 0\end{cases} $$ In **PReLU** the slope $\alpha$ stops being a constant and becomes a **learned parameter** - the network itself decides how much to let negative values through. It was from PReLU that the loud 2015 result came: 4.94% top-5 error on ImageNet, the first to exceed human level (5.1%) on that benchmark[^he]. **ELU** additionally pushes the mean activation closer to zero (an effect similar to normalization), which speeds up learning[^clevert].
### GELU, Swish and gated GLUs: what really sits inside LLMs Here we reach the functions that power today's language models. ReLU has a sharp "elbow" at zero - it is non-smooth. Newer activations smooth out that corner and, more importantly, introduce subtle **gating**: instead of hard-passing or zeroing the input, they weight it softly depending on its value. **GELU** (Gaussian Error Linear Unit), instead of asking "is $x$ positive?", asks "how much does $x$ stand out against the noise?" and passes it through proportionally[^hendrycks]. This is precisely the activation inside GPT-2 and GPT-3. **Swish** (also known as SiLU) is a related idea - a smooth interpolation between a line and ReLU, controlled by a single parameter[^ramachandran]. The biggest practical leap, however, came from **gated linear units** (GLU). The idea: split the signal into two linear projections, pass one of them through an activation and use it as a "gate" multiplying the other. From this lineage come **GEGLU** and **SwiGLU** - and this is not a curiosity but practice: in careful benchmarks (models matched by parameter count and compute) the GEGLU and SwiGLU variants gave the **best quality** of all the FFN activations tested[^shazeer].
📐 For the curious: GELU, Swish and the GLU family GELU weights the input by the probability that a normal variable does not exceed $x$ (where $\Phi$, capital "phi", is the cumulative distribution function of the normal distribution)[^hendrycks]: $$ \mathrm{GELU}(x) = x\,\Phi(x) = x\cdot\tfrac{1}{2}\Big[1+\mathrm{erf}\!\big(x/\sqrt{2}\big)\Big] \approx 0.5\,x\Big(1 + \tanh\!\Big[\sqrt{2/\pi}\,\big(x + 0.044715\,x^3\big)\Big]\Big) $$ Swish is $x$ multiplied by the sigmoid, with a parameter $\beta$ (beta) regulating the shape[^ramachandran]: $$ \mathrm{Swish}(x) = x\cdot\sigma(\beta x) $$ For $\beta=1$ this is exactly SiLU; as $\beta\to\infty$, Swish tends to ReLU - hence "smooth interpolation". The gated variants multiply element-wise (the symbol $\otimes$) the activated projection by the plain linear one[^shazeer]: $$ \mathrm{GEGLU}(x) = \mathrm{GELU}(xW)\otimes xV, \qquad \mathrm{SwiGLU}(x) = \mathrm{Swish}(xW)\otimes xV $$ The catch: GLUs have **three** weight matrices instead of two, so to keep the parameter count constant one reduces the hidden-layer width $d_{ff}$ by a factor of $\tfrac{2}{3}$[^shazeer]. That is why the mysterious "$\tfrac{2}{3}\cdot 4d$" shows up in the LLaMA descriptions.
## 🏭 What really sits inside GPT, LLaMA and Gemma Theory is theory, but which function won in practice? Here is the hard fact, straight from the papers and the source code of the creators: - **GPT-2 / GPT-3 (OpenAI): GELU.** Confirmed directly from the GPT-2 code - the `gelu` function is exactly the tanh approximation from Hendrycks and Gimpel's paper[^gpt2]. For GPT-3, the use of GELU is taken to be consistent with GPT-2 (the GPT-3 paper itself does not state it explicitly[^brown]). - **PaLM (Google, 2022): SwiGLU** - chosen because it "significantly improves quality over standard ReLU, GELU or Swish"[^chowdhery]. - **LLaMA (Meta, 2023): SwiGLU** - quoting the paper: "We replace the ReLU non-linearity by the SwiGLU activation function (...), we use a dimension of $\tfrac{2}{3}4d$ instead of $4d$ as in PaLM"[^touvron]. - **Gemma (Google DeepMind, 2024): GeGLU** - "The standard ReLU non-linearity is replaced by the GeGLU activation function"[^gemma]. The trend is clear: older OpenAI models stand on **GELU**, while the wave of modern, open LLMs (LLaMA, PaLM) switched to **SwiGLU**, whereas Gemma went for **GeGLU**. All of these choices are direct descendants of a single paper by Shazeer from 2020[^shazeer]. It is a good example of how a single, seemingly minor result - "multiply two projections instead of one" - seeps into an entire generation of models. ## 🎯 What's next We took the smallest building block fully apart: where it came from (McCulloch and Pitts' threshold unit), why it needs nonlinearity (because otherwise a stack of layers collapses into a single line), what that nonlinearity promises and what it does not (universal approximation - existence, not feasibility) and which concrete "bend" won in real models (the smooth, gated GELU and SwiGLU). Along the way we also saw the first crack in which the mystery of this series lives: our neuron is a caricature of biology, and yet out of its billions of copies grows something that the formula $y=\phi(\mathbf{w}\cdot\mathbf{x}+b)$ does not foretell with a single word. Understanding the building block takes nothing away from its remarkableness - it only shows how modest the raw material is. In the next stop of the series we take a handful of these neurons and arrange them into **layers and a network** - and then ask the most important question: where do those famous "billions of parameters" actually come from, and how does training turn random noise into a model that understands language? That is where backpropagation, loss functions and scaling laws will appear. If you would rather **touch** now instead of reading on, the [interactive "Anatomy of an LLM" page](/en/llm/anatomy#neuron) lets you drag the neuron's weights and switch activation functions live. And if you are curious about what actually *emerges* from this machinery once the building blocks number in the billions - [I wrote about that separately](/en/posts/rezonans-czlowiek-llm-kwantowe-lustro). [^mcculloch]: W. S. McCulloch, W. H. Pitts (1943). *A Logical Calculus of the Ideas Immanent in Nervous Activity*. The Bulletin of Mathematical Biophysics 5(4):115-133. [^rosenblatt]: F. Rosenblatt (1958). *The Perceptron: A Probabilistic Model for Information Storage and Organization in the Brain*. Psychological Review 65(6):386-408. [^minsky]: M. Minsky, S. Papert, *Perceptrons* (MIT Press, 1969) - the proof of the XOR limitation and the classic characterization of the artificial neuron as an idealization of the biological one (facts independently confirmed in the universal-approximation works). [^cybenko]: G. Cybenko (1989). *Approximation by Superpositions of a Sigmoidal Function*. Mathematics of Control, Signals and Systems 2:303-314. [^hornik]: K. Hornik (1991). *Approximation Capabilities of Multilayer Feedforward Networks*. Neural Networks 4(2):251-257. [^telgarsky]: M. Telgarsky (2016). *Benefits of Depth in Neural Networks*. COLT 2016, PMLR 49:1517-1539. [^hendrycks]: D. Hendrycks, K. Gimpel (2016). *Gaussian Error Linear Units (GELUs)*. arXiv:1606.08415. [^nair]: V. Nair, G. E. Hinton (2010). *Rectified Linear Units Improve Restricted Boltzmann Machines*. ICML-10, pp. 807-814. [^glorot]: X. Glorot, A. Bordes, Y. Bengio (2011). *Deep Sparse Rectifier Neural Networks*. AISTATS, PMLR 15:315-323. [^maas]: A. L. Maas, A. Y. Hannun, A. Y. Ng (2013). *Rectifier Nonlinearities Improve Neural Network Acoustic Models*. ICML Workshop. [^he]: K. He, X. Zhang, S. Ren, J. Sun (2015). *Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification*. arXiv:1502.01852. [^clevert]: D.-A. Clevert, T. Unterthiner, S. Hochreiter (2015). *Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)*. arXiv:1511.07289 (ICLR 2016). [^ramachandran]: P. Ramachandran, B. Zoph, Q. V. Le (2017). *Searching for Activation Functions* (Swish). arXiv:1710.05941. [^shazeer]: N. Shazeer (2020). *GLU Variants Improve Transformer*. arXiv:2002.05202. [^gpt2]: OpenAI. GPT-2 source code, `src/model.py`, the `gelu` function. [^brown]: T. B. Brown et al. (2020). *Language Models are Few-Shot Learners* (GPT-3). arXiv:2005.14165. [^chowdhery]: A. Chowdhery et al. (2022). *PaLM: Scaling Language Modeling with Pathways*. arXiv:2204.02311. [^touvron]: H. Touvron et al. (2023). *LLaMA: Open and Efficient Foundation Language Models*. arXiv:2302.13971. [^gemma]: Gemma Team, Google DeepMind (2024). *Gemma: Open Models Based on Gemini Research and Technology*. arXiv:2403.08295.