Neural Networks
Neural networks are at the heart of many modern technologies. Whether you are unlocking your phone with your face, being recommended a video to watch next, or are chatting with a large language model such as ChatGPT, there is a neural network at play behind the scenes. But how do they work? In this post we’ll look at a simple network, work out exactly how it operates, and teach it to do something useful.
1. The components of a neural network
A neural network is made out of four components:
-
A neuron is a node in the network, and it only does three simple things: (i) receive a bunch of input values, (ii) combine these inputs into a single value, and (iii) pass this result along further down the network. To do this, it relies on the other three network components.
-
A weight, , is the connection between two neurons. The weight determines how strongly an input contributes to the output value of the corresponding node.
-
A bias, , is the value of a node when it doesn’t receive any inputs. If the node does receive input, the bias simply offsets the output by a fixed amount.
-
An activation function, , is any non-linear function that inputs one number and outputs another, used at the end of a node’s calculation. Why? Without it, a neuron only multiplies and adds, such that a neural network could only perform linear operations (i.e, draw straight lines through your data). The activation function resolves this problem and allows the network to solve more complicated (non-linear) problems. Two popular choices:
- The sigmoid, , is an s-curve that squashes any number into the range 0-1. A welcome side effect is that this prevents values from exploding to infinity as they pass through many layers in the network.
- The ReLU (rectified linear unit), , simply sets negative numbers to zero. It is the default choice in modern networks, because it is fast to compute and makes training easier.
Putting these ingredients together, a neuron takes any number of inputs, , and computes a single output as
2. The forward pass
In the above illustration you can see a small network consisting of three layers. The input layer holds the data we feed in, the output layer holds the answer, and the layer in between is called a hidden layer. The weights and biases of this network are
where the superscript is used to denote a layer or an interface between two layers. A quick word on notation: I will use to denote the connection between the second node in the first layer to the first node in the second layer (i.e. -0.41). Together, the weights and biases serve as the parameters of the network; they are the values we should modify if we want to change how the network behaves. Let us now feed some data into the network:
The act of supplying these numbers into the network, calculating node values layer-by-layer, is called a forward pass. You can watch this process in action below:
Now you know exactly what neural network does. However, it may not be so clear how this could ever become useful, because the results produced by our current network are pretty meaningless. This is because the network parameters were chosen at random. To do something useful, we have to find just the right combination of weights and biases that allows the network to solve a problem of our interest. The process of finding such a useful set of parameters is called training.
3. Training a neural network
To train a neural network, we need three things:
- An objective. The problem we want to solve. What should our network be able to do?
- A metric for success. How well is our network currently performing?
- An optimization strategy. Which changes to the weights and biases could make the network better?
Let us go through these steps and train our network to do something useful.
3.1 Training objective - XOR
Our network is small, so let us give it a simple task. Using only 0 or 1 as possible inputs, do the following: If the two inputs are different, output 1; if the inputs are the same, output 0. You might recognize this task as the exclusive-or gate (XOR), and there are only four possible input combinations:
| XOR | ||
|---|---|---|
| 0 | 0 | 0 |
| 0 | 1 | 1 |
| 1 | 0 | 1 |
| 1 | 1 | 0 |
XOR is a suitable test-case, because it is one of the simplest problems that requires a non-linear solution. It is not obvious which set of weights and biases we need (feel free to try it), yet our tiny network is large enough to learn how to do it.
3.2 Metric for success/failure - Loss
Currently, out network is getting the answer wrong in all possible cases. To improve the network, we need a measure for how wrong it is. This number is called the loss, . There are different loss-metrics we could choose, but the simplest one is the squared error:
where is the network’s answer, in our case the value of node .
3.3 Optimization strategy - Backpropagation
So how do we tune our parameters in a way that improves the network? Remember, we are trying to minimize the loss or our network by tweaking its weights and biases. A useful analogy would be that you are searching for the valley of a misty landscape. Here, the height of the landscape resembles the loss. The weights and biases are like the different directions you could take to explore the landscape. At the valley, a step in any direction would increase the loss of the network and thus make it worse, hence this point marks our solution. So how do you find this valley? Since you initially have no way of telling where it is, your best bet is to just keep walking in the steepest downhill direction. Writing this idea in mathematical terms: we should update our weights as
where the learning rate, , is the number of steps you take before reevaluating your direction. The minus sign just means go downhill, not uphill. One key question remains: how do we find this slope, ? This is done through a process called backpropagation. The central idea behind backpropagation is pretty simple: keep track of how error propagates through the network () so that we can reduce the error. The simulation below should help illustrate the basic idea. The mathematics involved are a little bit more technical, however, so don’t worry about the precise calculations right away - I continue to explain them below.
3.3.1 Backpropagation - Updating weights
Our task is to figure out how changes in weight influence the loss: . However, the weights don’t influence the loss directly. Instead, we are dealing with the following chain of influence:
We need to take into account how error propagates through each step. The chain rule can be used to find the slope:
Hence, we compute how errors propagate through each link in the chain and take their product to find the overall slope. The derivatives of the individuals steps are not too hard to work out; they are summarized below.
| Function | Derivative | ||
|---|---|---|---|
Now, try to view the animation again and see if you can follow along. In the first step, the network answered , where the target was . This tells us
Secondly, we account for the activation function by multiplying with its slope, .
Lastly, we account for the weight, . For the top connection into the output node, we have . Therefore
The remainder of the animation just keeps repeating this same trick until a value is obtained for every . All that remains is to apply the update rule. For example, for the top connection into the output node ():
3.3.2 Backpropagation - Updating biases
Updating the biases isn’t much different. Their chain of influence is:
So the chain rule gives
Only the last factor is new. From the definition :
Which basically means we can skip the last step: matches the purple values in our simulation.
3.4 A fully trained neural network
You could see in the backpropagation animation that with each round the loss got smaller and the network got closer to the correct answer. I took the liberty of performing the above-shown process for 650,000 rounds. Feel free to play around with it, and see whether it has learned XOR:
Thats it for now. A modest victory, but do not underestimate the potential. In the next post, we’ll explore how the simple act of scaling up allows us to do far more challenging tasks. Catch you in the next one!
-Kasper