Activation Functions: Key to Neural Network Intelligence

The Importance of Activation Functions in Neural Networks

Neural networks, when built with many linear layers without non-linear activation functions, become incapable of performing complex tasks. Indeed, each linear layer performs an affine transformation, which boils down to a matrix multiplication followed by a vector addition. Mathematically, this is expressed by the equation y = Wx + b. If we stack several of these layers, the network reduces to a simple affine transformation, regardless of the number of layers involved. Thus, a network with ten, one hundred, or even one thousand linear layers simplifies to a single operation: output = Wₑ · x + bₑ.

To better understand, let's examine the equation:

• Layer 1: y₁ = W₁x + b₁
• Layer 2: y₂ = W₂y₁ + b₂
• Layer 3: y₃ = W₃y₂ + b₃

By inserting each layer into the next, we obtain: y₃ = W₃(W₂(W₁x + b₁) + b₂) + b₃. When multiplied, this becomes: y₃ = (W₃W₂W₁)x + (W₃W₂b₁ + W₃b₂ + b₃).

Rather than manipulating these nested matrices, we can group them into two variables: Wₑ = W₃W₂W₁, the global weight matrix, and bₑ = W₃W₂b₁ + W₃b₂ + b₃, the global bias vector. The three-layer network then reduces to a single layer: output = Wₑ · x + bₑ.

The Necessity of Non-Linearity

Why is it crucial for a network to deviate from this linearity? Imagine a map where a small island is surrounded by the ocean. If you were asked to draw a straight line to separate the island from the ocean, it would be impossible. A straight line would inevitably cross both the island and the ocean. What we need is a non-linear boundary that can encircle the island and distinguish it from the ocean. This is exactly what neural networks attempt to achieve. Instead of being limited to linear separations, they learn to transform data into representations where complex boundaries become possible.

Thus, when a neural network "separates datasets," it means it learns a decision function that divides the input space into distinct regions: on one side, everything belongs to class A, and on the other, everything belongs to class B. The nature of this boundary, whether straight, curved, or complex, entirely depends on the arrangement of the data.

The Crucial Role of Activation Functions

Activation functions play an essential role by introducing non-linearity into neural networks. After each linear transformation in a layer, an activation function is applied to transform the output before it moves to the next step. For example, instead of z = W₂(W₁x + b₁) + b₂, the result becomes z = W₂·f(W₁x + b₁) + b₂, where f is a non-linear function such as sigmoid, tanh, or ReLU. Even a simple function like ReLU, defined as max(0, x), is sufficient to prevent the network from collapsing into a simple linear transformation.

This simple function f breaks the linearity: there is no matrix M and vector c such that f(W₁x + b₁) = Wx + b for every x. By stacking enough of these non-linear steps, the network transcends simple linear thinking and can create complex shapes like circles and spirals. This is the essence of an activation function: it allows a network to move from a simple ability to draw straight lines to the possibility of encircling complex shapes.

Each neuron contributes individually to a slight curvature, and a network contains thousands of them, each adding curvature in a different location. By stacking enough neurons, the network can approximate curves and boundaries that even a single neuron, or even a hundred, could not achieve alone.

Contextual Interpretation in Language Models

Let's take the example of an ambiguous sentence: "Time flies like an arrow; fruit flies like a banana." In the first part, "flies" is a verb indicating that time passes quickly. In the second, "flies" is a noun, referring to a type of insect. The surrounding context is what allows us to distinguish these two usages.

A language model must be able to separate these two meanings into different regions of its internal representation, even though, at the input level, it is the same word. This is where the depth and non-linearity of neural networks become crucial. Each layer starts from a different random point and ultimately draws its own curved boundary through the data. As training progresses, this initial divergence transforms into more precise specialization.

Although this example is simplified, in reality, models do not neatly divide layers into "this layer = nouns, this layer = verbs." However, one can imagine that the initial layers capture local patterns, such as word order or part of speech, while deeper layers integrate more context and begin to represent more abstract concepts, such as the meaning of "flies" or contextual references. It is this progressive curvature, layer by layer, that allows the network to distinguish "time flies" from "fruit flies" in different representation spaces.

The Universal Approximation Theorem

The Universal Approximation Theorem (UAT) states that a feedforward neural network with a non-linear activation function and a sufficiently large hidden layer can, in theory, approximate any continuous function over a bounded domain with the desired precision. This capability can be understood by imagining how networks combine many simple non-linear components to create increasingly complex shapes and behaviors. These components can be...