The beginning of Neurocomputing dates back to 1943 when McCulloch and Pitts showed that even simple types of neural networks could, in principle, compute any arithmetic or logic function [1]. The Backpropagation Algorithm, which is a major tool in minimizing error in ANN, was first proposed by Werbos (1974) [2], and since then has been employed in a number of fields for solving problems that would be quite difficult using conventional computer science techniques. In Nielson (2023) [3], it is shown that it is possible to adjust the weight of a multilayer feed-forward neural network in a systematic way to learn the implicit mapping in a set of input-output pattern pairs. The learning law is called the generalized delta rule or error backpropagation.

This manuscript does not propose a new algorithm or a theoretical advance. Instead, it serves as an educational resource in the form of a fully explicit, manually verified walkthrough of backpropagation with accompanying MATLAB code. All mathematical derivations follow classical references [2] and [3], but the step-by-step numerical presentation with all intermediate values is intended to reduce the learning curve for students and practitioners.

Multilayer perceptrons (MLP) are the most popular type of neural networks [4]. They belong to a general class of structures called feedforward neural networks, a basic type of neural network capable of approximating generic classes of functions, including continuous and integrable functions. MLP neural networks have been used in a variety of microwave modelling and optimization problems.

The Multilayer Perceptron (MLP) neural network structure is depicted below in Figure 1, incorporating the input layer, hidden layers and output layer.

Figure 1. General neural network (Multilayer perceptron).

This particular model can accommodate many inputs ( x 1 , x 2 , ⋯ , x n ) , and the outputs ( y 1 , y 2 , ⋯ , y n ) can each be related to the inputs in the following way, y i = ϕ i ( x 1 , x 2 , ⋯ , x n ) , i = 1 , ⋯ , n , where the ϕ i ’s are the respective activation functions.

Activation functions are very important in Machine Learning because they incorporate nonlinearity into our ANN [2]. This enables the neural network to learn complicated tasks; without it, the neural network can only perform linearregression. The most popular hidden neuron activation functions are the sigmoid, arctangent, and the hyperbolic tangent functions given respectively by (Figure 2).

(a)

(b) (c)

Figure 2. Activations functions. (a) σ 1 ( x ) , (b) σ 2 ( x ) , (c) σ 3 ( x ) .

The sigmoid function σ 1 ( x ) is a smooth switch function having the following property;

All these logistic functions are bounded, continuous, monotonic, and continuously differentiable.

In this work, we require the derivative of the sigmoid function. Thus given

We have

Now

The exact contribution of this paper is a fully worked numerical example of backpropagation for a 3-2-2 network (One input layer with three neurons, one hidden layer with two neurons and one output layer with two neurons, Figure 6), including all intermediate gradient calculations, followed by a verifiable MATLAB implementation. No novel algorithmic modifications are proposed; instead, we prioritize transparency and reproducibility for educational purposes.

Let I n represent an n-dimensional unit cube containing all possible input samples x = ( x 1 , x 2 , ⋯ , x n ) with x i ∈ [ 0 , 1 ] , i = 1 , 2 , ⋯ , n . Let C ( I n ) be the space of continuous functions on I n , given a continuous sigmoid function σ ( ⋅ ) , then universal approximation theorem states that the finite sums of the form

are dense in C ( I n ) . This simply means that given any function f ∈ C ( I n ) and ε > 0 , there is a sum y ( x , w ) of the above form that satisfies

The universal approximation theorem expressed in Equation (4) simply states that there always exists a 3-layer perceptron that can approximate an arbitrary nonlinear, continuous, multidimensional function f with any desired accuracy. However, the theorem does not state how many neurons are needed by the three-layer MLP to approximate the given function; as such, failure to develop an accurate model can be attributed to an inadequate number of hidden neurons, inadequate learning or training, or perhaps the presence of a stochastic rather than a deterministic relation between inputs and outputs [5].

The computational structure of a neuron (Figure 3) consists of a set of synapses, each of which is characterized by

Figure 3. Computational structure of a neuron.

A weight or strength of its own.An adder, an activation function and a bias.

In mathematical terms, a neuron k can be described by:

where u k is the linear combiner output due to input signals. Moreover, v k = u k + b k .

The bias is an external parameter of artificial neuron and can be included into Equation (6) as follows:

We consider a Multilayer Perceptron Feed Forward Neural Network with binary inputs. The network diagram is given below (Figure 4).

Figure 4. MLP Neural Network with one hidden layer containing two neurons H 1 , H 2 .

In the above network diagram I 1 , I 2 are the inputs, B 1 , B 2 the bias, H 1 , H 2 are the two neurons in the single hidden layer and O 1 is the output. The distribution of weights is given in Table 1. Furthermore, we employ as activation function the sigmoid function

Table 1.Weight distribution.

The computations are as follows:

The final output of the multilayer network is thus 0.989.

All neural networks must be trained, and the common training algorithm is depicted in the flowchart below (Figure 5).

Figure 5. Network training (One training iteration).

Backpropagation actually stands for backwardpropagationoferrors, and is commonly used technique for training neural network. It is a mechanism employed to update the weights using gradientdescent. It computes the gradient of the error function with respect to the neural network’s weights. The calculation proceeds backwards [5] through the network. We note that gradient descent is an iterative optimization algorithm for finding the minimum of a function; in our case we want to minimize the error function [6]. To find a local minimum of a function using gradient descent, one takes steps proportional to the negative of the gradient of the function at the current point.

The Backpropagation algorithm is a supervised learning method for multilayer feed-forward neural networks. Backpropagation is the algorithm that runs deep learning [7].

The principle of the backpropagation approach is to model a given function by modifying internal weights of input signals to produce an expected output signal. The system is trained using a supervised learning method, where the error between the system’s output and a known expected output is presented to the system and used to modify its internal state. Technically, the backpropagation algorithm is a method for training the weights in a multilayer feed-forward neural network. As such, it requires a network structure to be defined of one or more layers where one layer is fully connected to the next layer. A standard network structure is one input layer, one hidden layer, and one output layer. In the following demonstration, we use a single training example ( x 1 = 1 , x 2 = 4 , x 3 = 5 ) with target outputs ( t 1 = 0.1 , t 2 = 0.05 ). The network is not intended for generalization but to illustrate one complete forward-backward-update cycle.

Choice of Hyperparameters and Activation Function

The sigmoid activation is chosen because it is differentiable, bounded, and produces a simple derivative σ ′ ( x ) = σ ( x ) ( 1 − σ ( x ) ) , which simplifies manual calculations. The sum-of-squares error (SSE) is standard for regression-like tasks and yields clean gradient expressions. A learning rate of 0.01 is sufficiently small to avoid divergence but large enough to show parameter changes within a few decimal places. Initial weights are arbitrarily selected small numbers (0.1 to 0.9) to avoid saturation of the sigmoid at the start. The iteration count of 10,000 ensures the error has visibly converged; this is empirically determined from the MATLAB output.

In this section, we consider an example of one iteration of the backpropagation algorithm employing appropriate formulae from basic principles and actual values. The neural network will consist of three input neurons, one hidden layer with two neurons, and an output layer with two neurons (Figure 6).

Figure 6. Schematic for the Network training (One training iteration).

We will take into consideration the following important steps.

1) Initialize weights for the parameters we want to train.

2) Forward propagate through the network to get the output values.

3) Define the error or cost function and its first derivatives.

4) Backpropagate through the network to determine the error derivatives.

5) Update the parameter estimates using the error derivative and the current value.

The input and target values for this problem are x 1 = 1 , x 2 = 4 , x 3 = 5 and t 1 = 0.1 , t 2 = 0.05 . We initialize the weights as depicted in the diagram below. In general, we assign them randomly for illustration purposes. We chose the numbers as shown (Figure 7).

Figure 7. Schematic with parameter values of the network (one training iteration).

Mathematically, we have the following relationships between nodes in the networks. For the input and output layer, we prefer the convention of setting z h 1 , z h 2 , z o 1 , z o 2 to denote the values before the activation function is applied and the notation of h 1 , h 2 , o 1 , o 2 to denote the values after application of the activation function.

Inputtohiddenlayer

Hiddenlayertooutputlayer

We can use Equations (8) and (9) above to forward propagate through the network. We exhibit up to four decimal places below, but maintain all decimals in actual calculations.

We now define the sum of squares error (SSE) using the target values and the results from the last layer from forward propagation.

We can now backpropagate through the network to compute all the error derivatives with respect to the parameters. Note that in what follows we employ only the basic principles of calculus, which is the chainrule or function of a function rule. Furthermore, given that w 7 h 1 + w 9 h 2 + b 2 = z o 1 and w 8 h 1 + w 10 h 2 + b 2 = z o 2 , we have the simple partial derivatives;

We are now ready to compute ∂ E / ∂ w 7 , ∂ E / ∂ w 8 , ∂ E / ∂ w 9 and ∂ E / ∂ w 10 employing our previous derivatives; we have

Similarly, we have;

The error derivative of b 2 is a little bit more involved since changes to b 2 affect the error through both o 1 and o 2 .

In summary, we have computed numerical values for the error derivatives with respect to w 7 , w 8 . w 9 , w 10 , and b 2 . We will now back-propagate one layer to compute the error derivatives of the parameters connecting the inputlayer to the hiddenlayer. These error derivatives are seven in number, namely;

We will compute ∂ E / ∂ w 1 , ∂ E / ∂ w 3 and ∂ E / ∂ w 5 first since they all flow through the h 1 node.

The computation of the first term on the right-hand side of the equation above is a bit more involved than previous calculations since h 1 affects the error through both o 1 and o 2 . As usual chain rule, we have;

Now, we proceed with the numerical values for the error derivatives above. These derivatives have already been computed above. Substituting the above into the formula for ∂ E / ∂ w 1 , we obtain;

The calculations for ∂ E / ∂ w 3 and ∂ E / ∂ w 5 are as follows

We now compute ∂ E / ∂ w 2 , ∂ E / ∂ w 4 and ∂ E / ∂ w 6 since they all flow through the h 2 node. We have;

The computation of the first term on the right hand side of the equation above is a bit more involved since h 2 affects the error through both o 1 and o 2 .

Substituting the above into Equation (17) for ∂ E / ∂ w 2 we obtain

The computations for ∂ E / ∂ w 4 and ∂ E / ∂ w 6 are as follows;

The final error derivative we have to compute is ∂ E / ∂ b 1 , which is done as follows;

We now have computed all error derivatives and now proceed to make the parameter updates after the first iteration of backpropagation. We employ the learning rate of α = 0.01 . Thus;

We remark that the quantities ∂ E / ∂ w j j = 1 , 2 , ⋯ , 10 and ∂ E / ∂ b i i = 1 , 2 are respectively the rate at which the error E is changing with respect to the weights w j and the biases b i . Having done all the above computations, it is necessary to repeat them many times until the error (SSE) decreases and the parameter estimates stabilize or converge to some values, as shown in Figure 8. This task, however, cannot be achieved manually as it is computationally intensive. We will therefore rely on a MATLAB program to execute these complex iterations for speed and accuracy. The MATLAB code is given in the Appendix below.

Figure 8. Sum of squares error after 10,000 iterations.