In this study we investigate neural network solutions to nonlinear differential equations of Ricatti-type. We employ a feed-forward Multilayer Perceptron Neural Network (MLPNN), but avoid the standard back-propagation algorithm for updating the intrinsic weights. Our objective is to minimize an error, which is a function of the network parameters i.e., the weights and biases. Once the weights of the neural network are obtained by our systematic procedure, we need not adjust all the parameters in the network, as postulated by many researchers before us, in order to achieve convergence. We only need to fine-tune our biases which are fixed to lie in a certain given range, and convergence to a solution with an acceptable minimum error is achieved. This greatly reduces the computational complexity of the given problem. We provide two important ODE examples, the first is a Ricatti type differential equation to which the procedure is applied, and this gave us perfect agreement with the exact solution. The second example however provided us with only an acceptable approximation to the exact solution. Our novel artificial neural networks procedure has demonstrated quite clearly the function approximation capabilities of ANN in the solution of nonlinear differential equations of Ricatti type.
We present a new perspective for obtaining solutions of initial value problems of Ricatti-type [
A neural network is an inter-connection of processing elements, units or nodes, whose functionality resemble that of the human neurons. The processing ability of the network is stored in the connection strengths, simply called weights, which can be obtained by a process of adaptation to, a set of training patterns. Neural network methods can solve both ordinary and partial differential equations. Furthermore, it relies on the function approximation property of feed forward neural networks which results in a solution written in a closed analytic form. This form employs a feed forward neural network as a basic approximation element. Training of the neural network can be done either by any optimization technique which in turn requires the computation of the gradient of the error with respect to the network parameters, by regression based model or by basis function approximation.
Artificial neural network can make a nonlinear mapping from the inputs to the outputs of the corresponding system of neurons which is suitable for analyzing the problem defined by initial/boundary value problems that have no analytical solutions or which cannot be easily computed. One of the applications of the multilayer feed forward neural network is the global approximation of real valued multivariable function in a closed analytic form. Namely such neural networks are universal approximators. It has been find out in the literature that multilayer feed forward neural networks with one hidden layer using arbitrary squashing functions are capable of approximating any Borel measurable function from one finite dimensional space to another with any desired degree of accuracy. This is made clear in the following theorem.
The universal approximation theorem for MLP was proved by Cybenko [
y k = y k ( x , w ) = ∑ i = 1 N 2 w k i 3 φ ( ∑ j = 0 n w k i 2 x j ) , k = 1 , 2 , ⋯ , m (1)
are dense in C ( I n ) . This simply means that given any function f ∈ C ( I n ) and ε > 0 , there is a sum y ( x , w ) of the above form that satisfies
| y ( x , w ) − f ( x ) | < ε , ∀ x ∈ I n . (2)
A neural network has to be configured such that the application of a set of inputs produces the desired set of outputs. Various methods to set the strengths of the connection exist. One way is to set the weights explicitly, using priory knowledge. Another way is to train the neural network by feeding it, teaching patterns and letting it change its weights according to some learning rule. The term learning is widely used in the neural network field to describe this process; it might be formally described as: determining an optimized set of weights based on the statistics of the examples. The learning classification situations in neural networks may be classified into distinct sorts of learning: supervised learning, unsupervised learning, reinforcement learning and competitive learning [
Next step is to compute the gradient with respect to input vectors, for this purpose let us consider a multilayer perceptron (MLP) neural network [
N ( x , p ) = ∑ i = 1 m v j φ ( z j ) , z j = ∑ i = 1 n w j i x i + u j . (3)
w j i denotes the weight from input unit 𝑖 to the hidden unit 𝑗, v j denotes weight from the hidden unit 𝑗 to the output unit, u j denotes the biases, and φ ( z j ) is the sigmoid activation function.
Now the derivative of networks output N with respect to input vector x i is:
∂ ∂ x i N ( x , p ) = ∂ ∂ x i ( ∑ j = 1 m v j φ ( z j ) ) = ∑ j = 1 m v j w j i φ ( 1 ) (4)
where φ ( 1 ) ≡ ∂ φ ( x ) / ∂ x . Similarly, the kth derivative of N is computed as; ∂ k N / ∂ x i k = ∑ j = 1 m v j w j i k φ j ( k )
Where φ j ≡ φ ( z j ) and φ ( k ) denotes the kth order derivative of the sigmoid activation function.
Let us consider the following general differential equations which represent both ordinary and partial differential equations Majidzadeh [
G ( x , ψ ( x ) , ∇ ψ ( x ) , ∇ 2 ψ ( x ) , ⋯ ) = 0 , ∀ x ∈ D , (5)
subject to some initial or boundary conditions, where x = ( x 1 , x 2 , ⋯ , x n ) ∈ ℝ n , D ⊂ ℝ n denotes the domain, and ψ ( x ) is the unknown scalar-valued solution to be computed. Here, G is the function which defines the structure of the differential equation and ∇ is a differential operator. Let ψ t ( x , p ) denote the trail solution with parameters (weights, biases) p. Legaris et al. [
ψ t ( x , p ) = A ( x ) + F ( x , N ( x , p ) ) (6)
where A ( x ) satisfies initial or boundary condition and contains no adjustable parameters, whereas N ( x , p ) is the output of feed forward neural network with the parameters p and input data x. The function F ( x , N ( x , p ) ) is actually the operational model of the neural network. Feed forward neural network (FFNN) converts differential equation problem to function approximation problem. The neural network N ( x , p ) is given by
N ( x , p ) = ∑ j = 1 m v j σ ( z j ) , z j = ∑ i = 1 n w j i x i + u j . (7)
w j i denotes the weight from input unit 𝑖 to the hidden unit j, v j denotes weight from the hidden unit j to the output unit, u j denotes the biases, and σ ( z j ) is the sigmoid activation function.
The neural network weights determine the closeness of predicted outcome to the desired outcome. If the neural network weights are not able to make the correct prediction, then only the biases need to be adjusted. The basis function we shall apply in this work in training the neural network is the sigmoid activation function given by
σ ( z j ) = ( 1 + e − z j ) − 1 . (8)
Let us consider the first order ordinary differential equation below
ψ ′ ( x ) = f ( x , ψ ) , x ∈ [ a , b ] (9)
with initial condition ψ ( a ) = A . In this case we assume the function f is nonlinear in its argument. The ANN trial solution may be written as
ψ t ( x , p ) = A + x N ( x , p ) , (10)
where N ( x , p ) is the neural output of the feed forward network with one input data x with parameters p. The trial solution ψ t ( x , p ) satisfies the initial condition. To solve this problem using neural network (NN), we shall employ a NN architecture with three layers. One input layer with one neuron; one hidden layer with n neurons and one output layer with one output unit, as depicted in
Each neuron is connected to other neurons of the previous layer through adaptable synaptic weights w 1 j and biases u j . Now, ψ t ( x i , p ) = A + x i N ( x i , p ) with
N ( x , p ) = ∑ j = 1 n v j σ ( x w j + u j ) , z j = x w j + u j . (11)
It is possible to have Multi-layered perceptrons with more than three layers, in which case we have more hidden layers [
σ ( z ) = ( 1 + e − z ) − 1 (12)
with the property that;
σ ′ ( z ) = σ ( z ) ( 1 − σ ( z ) ) . (13)
The trial solution ψ t ( x , p ) satisfies the initial condition. We differentiate the trial solution ψ t ( x , p ) to get
d ψ t ( x , p ) d x = N ( x , p ) + x d N ( x , p ) d x , (14)
We observe that;
d N ( x , p ) d x = ∑ j = 1 n v j d d x σ ( x w j + u j ) = ∑ j = 1 n v j w j σ ′ ( z j )
⇒ d N ( x , p ) d x = ∑ j = 1 n v j w j σ ( z j ) ( 1 − σ ( z j ) )
For evaluating the derivative term in the right hand side of (32), we use equations (7) and (26)-(31).
The error function for this case is formulated as;
E ( p ) = ∑ i = 1 n ( d ψ t ( x i , p ) d x i − f ( x i , ψ t ( x i , p ) ) ) 2 . (15)
Minimization of the above error function is considered as a procedure for training the neural network, where the error corresponding to each input vector x is the value f ( x ) which has to become zero. In computing this error value, we require the network output as well as the derivatives of the output with respect to the input vectors. Therefore, while computing error with respect to the network parameters, we need to compute not only the gradient of the network but also the gradient of the network derivatives with respect to its inputs [
The Riccati equation is a nonlinear ordinary differential equation of first order of the form:
y ′ ( x ) = p ( x ) y + q ( x ) y 2 + r ( x ) (16)
where p ( x ) , q ( x ) , r ( x ) are continuous functions of x. Neural network method can also solve this type of ODE. We show how our new approach can solve this type of ODE by redefining the neural network with respect to the form the ODE takes. Specifically, we consider the initial value problem:
y ′ ( x ) = 2 y ( x ) − y 2 ( x ) + 1 , y ( 0 ) = 0 , x ∈ [ 0 , 1 ] , (17)
which was solved by Otadi and Mosleh (2011) [
The trial solution is given by y t ( x ) = A + x ℵ ( x , p ) . Applying the initial conditions gives A = 0 . Therefore y t ( x ) = x ℵ ( x , p ) . This solution obviously satisfies the given initial condition. We observe that in Equation (17), the term y 2 ( x ) is what makes the ODE nonlinear. Also this term cannot be separated from 2 y ( x ) . Therefore, we incorporate 2 y ( x ) − y 2 ( x ) into the neural network to take care of the nonlinearity seen in the given differential equation. Thus, the new neural network becomes,
ℵ ( x , p ) = ∑ j m v j [ 2 σ ( z j ) − σ 2 ( z j ) ] = ∑ j m v j σ ( z j ) [ 2 − σ ( z j ) ] (18)
The error to be minimized is
E = 1 2 ∑ i = 1 n { d d t y t ( x i , p ) − [ 2 y t ( x i , p ) − y t 2 ( x i , p ) + 1 ] } 2 (19)
where the set { x i , i = 1 , ⋯ , n } are the discrete points in the interval [ 0 , 1 ] . We proceed as follows.
To compute the weights w j , j = 1 , 2 , 3 from the input layer to the hidden layer (
[ w 1 w 2 w 3 ] = [ φ 1 ( x 1 ) φ 2 ( x 1 ) φ 3 ( x 1 ) φ 1 ( x 2 ) φ 2 ( x 2 ) φ 3 ( x 2 ) φ 1 ( x 3 ) φ 2 ( x 3 ) φ 3 ( x 3 ) ] − 1 [ f 1 f 2 f 3 ] (20)
Here;
φ i ( x ) = exp ( − | x − x i | 2 2 σ 2 ) , σ 2 = 1 N ∑ i = 1 N ( x i − x ¯ ) 2 , x ¯ = 1 N ∑ i = 1 N x i (21)
The above is the so-called Gaussian Radial Basis function (GRBF) approximation model. To obtain the weights ν j , j = 1 , 2 , 3 from hidden layer to the output layer, we construct another function θ ( x ) such that ν = ϕ − 1 f , where, f ( x ) = ( θ ( x 1 ) , θ ( x 2 ) , θ ( x 3 ) ) T , x = ( x 1 , x 2 , x 3 ) and ϕ is given in Equation (20). We only need to replace the w j ’s by the ν j ’s, j = 1 , 2 , 3 .
The exact form of f ( x ) depends on the nature of a given differential equation. This will be made clear below. The nonlinear differential Equation (17) is rewritten as; y ′ ( x ) − 2 y ( x ) + y 2 ( x ) = 1 .
We now form a linear function based on the default sign of the differential equation, i.e. ϑ ( x ) = a x − b , where a is the coefficient of the derivative of y and b is the coefficient of y (i.e. a = 1 , b = − 2 ). Thus;
ϑ ( x ) = x + 2 , f ( x ) = ( ϑ ( x 1 ) , ϑ ( x 2 ) , ϑ ( x 3 ) ) T = ( 2.1 , 2.2 , 2.3 ) T , for x = ( 0.1 , 0.2 , 0.3 ) T .
This we apply to get the weights from input layer to the hidden layer. Thus f = ( 2.1 , 2.2 , 2.3 ) T and w = ϕ − 1 f
⇒ [ w 1 w 2 w 3 ] = [ 1 0 .94 0 .78 0 .94 1 0 .94 0 .78 0 .94 1 ] − 1 [ 2.1 2.2 2.3 ] (22)
Hence, the weights from the input layer to the hidden layer are
[ w 1 w 2 w 3 ] = [ 41 .335 − 73 .437 36 .79 − 73 .437 139 .062 − 73 .437 36 .79 − 73 .437 41 .335 ] [ 2.1 2.2 2.3 ] , [ w 1 w 2 w 3 ] = [ 9.858 − 17.187 10.767 ] (23)
The weights from input layer to the hidden layer are: w 1 = 9.858 , w 2 = − 17.187 , w 3 = 10.767 .
In order to get the weights from the hidden layer to the output layer, we now apply the forcing function which in this case is a constant function. That is, θ ( x ) = 1 , which is a constant function.
⇒ f ^ = ( θ ( x 1 ) , θ ( x 2 ) , θ ( x 3 ) ) T = ( 1 , 1,1 ) T (24)
θ ( x ) being the nonhomogeneous term. With v = ϕ − 1 f ^ the weights from the hidden layer to the output layer are given by
[ v 1 v 2 v 3 ] = [ 1 0 .94 0 .78 0 .94 1 0 .94 0 .78 0 .94 1 ] − 1 [ 1 1 1 ] = [ 41.335 − 73.437 36.79 − 73.437 139 .067 − 73.437 36.79 − 73.437 41 .335 ] [ 1 1 1 ]
⇒ [ v 1 v 2 v 3 ] = [ 4.687 − 7.812 4.687 ] (25)
Thus the weights from the hidden layer to the output layer are: v 1 = 4.687 , v 2 = − 7.812 , v 3 = 4.687 .
The biases are fixed between −20 and 20. We now train the network with the available parameters using our MathCAD 14 [
w 1 : = 9.858 w 2 : = − 17.187 w 3 : = 10.767 x : = 1 v 1 : = 4.687 v 2 : = − 7.812 v 3 : = 4.687 u 1 : = − 20 u 2 : = 10 u 3 : = − 12.534 z 1 : = w 1 ⋅ x + u 1 = − 10.142 z 2 : = w 2 ⋅ x + u 2 = − 7.187 z 3 : = w 3 ⋅ x + u 3 = − 1.767 σ ( z 1 ) : = [ 1 + exp ( z 1 ) ] − 1 = 3.9388 × 10 − 5 ,σ ( z 2 ) : = [ 1 + exp ( z 2 ) ] − 1 = 7.5578 × 10 − 4 , σ ( z 3 ) : = [ 1 + exp ( z 3 ) ] − 1 = 0.1459 ℵ : = v 1 ⋅ σ ( z 1 ) ⋅ ( 2 − σ ( z 1 ) ) + v 2 ⋅ σ ( z 2 ) ⋅ ( 2 − σ ( z 2 ) ) + v 3 ⋅ σ ( z 3 ) ⋅ ( 2 − σ ( z 3 ) ) = 1.256457 y p ( x ) : = x ⋅ ℵ = 1.256457 , y d ( x ) : = 2 ⋅ tanh ( x ⋅ 2 ) = 1 .256367 E : = 0.5 ⋅ ( y d ( x ) − y p ( x ) ) 2 = 4.05 × 10 − 4
The plots of the exact and predicted values in
We consider the initial value problem:
x 2 y ′ + x 2 y 2 = 2 , y ( 1 2 ) = 0 , x ∈ ( 0 , 1 ] (26)
The exact solution is easily computed as: y ( x ) = ( 8 x 3 − 1 ) ( x + 4 x 4 ) − 1 .
Our trial solution for the given problem is y t ( x ) = A + x ℵ ( x , p ) . Applying the initial conditions gives
A = − 1 2 ℵ ( 1 2 , p ) . Therefore, y t ( x ) = − 1 2 ℵ ( 1 2 , p ) + x ℵ ( x , p ) (27)
In Equation (26), the nonlinear term y 2 ( x ) is alone in the ode (i.e. dividing
| Input data (X) | 0 | 0.1 | 0.2 | 0.3 | 0.4 | 0.5 | 0.6 | 0.7 | 0.8 | 0.9 | 1 |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Y Exact | 0 | 0.19868 | 0.38967 | 0.56642 | 0.72434 | 0.86106 | 0.97623 | 1.07104 | 1.14761 | 1.20852 | 1.25637 |
| Y Pred | 0 | 0.19867 | 0.38967 | 0.56642 | 0.72433 | 0.86106 | 0.97622 | 1.07103 | 1.14764 | 1.20849 | 1.25639 |
out rightly by x 2 ). Therefore, our neural network for this problem takes the form:
ℵ ( x , p ) = ∑ j 3 v j σ 2 ( z j ) = ∑ j 3 v j σ ( z j ) [ σ ( z j ) ] (28)
We form algebraic equation of degree one with the default sign of the ode. Thus ϑ ( x ) = a x + b , ( a = x 2 , b = 0 ). Hence ϑ ( x ) = x 3 ⇒ f ( x ) = ( 0.001 , 0.008 , 0.027 ) T , for x = ( 0.1 , 0.2 , 0.3 ) T
This we apply to get the weights from input layer to the hidden layer. We employ the GRBF here for the weights w = ϕ − 1 f . Hence;
[ w 1 w 2 w 3 ] = [ 1 0 .94 0 .78 0 .94 1 0 .94 0 .78 0 .94 1 ] − 1 [ 0.001 0.008 0.027 ] ⇒ [ w 1 w 2 w 3 ] = [ 0.447 − 0.944 0.565 ] (29)
The weights from input layer to the hidden layer are: w 1 = 0.447 , w 2 = − 0.944 , w 3 = 0.565 .
We now use the forcing function, a constant function in this case, to get the weights from the hidden layer to the output layer. That is, θ ( x ) = 2 ⇒ f ^ ( x ) = ( 2 , 2 , 2 ) T for x = ( 0.1 , 0.2 , 0.3 ) T . Hence, the weights v = ϕ − 1 f ^ from the hidden layer to the output layer are;
[ v 1 v 2 v 3 ] = [ 1 0 .94 0 .78 0 .94 1 0 .94 0 .78 0 .94 1 ] − 1 [ 2 2 2 ] ⇒ [ v 1 v 2 v 3 ] = [ 9.375 − 15.625 9.375 ] (30)
The weights from the hidden layer to the output layer are: v 1 = 9.375 , v 2 = − 15.625 , v 3 = 9.375 .
The biases are fixed between −10 and 10. We now train the network with the available parameters using our MathCAD 14 algorithm as follows:
w 1 : = 1.234 w 2 : = − 2.725 w 3 : = 1.716 x : = 1 v 1 : = 9.375 v 2 : = − 15.625 v 3 : = 9.375 u 1 : = − 7 u 2 : = − 4 u 3 : = − 7 z 1 : = w 1 ⋅ x + u 1 = − 5.766 z 2 : = w 2 ⋅ x + u 2 = − 6.725 z 3 : = w 3 ⋅ x + u 3 = − 5.284
σ ( z 1 ) : = [ 1 + exp ( z 1 ) ] − 1 = 0.998 , σ ( z 2 ) : = [ 1 + exp ( z 2 ) ] − 1 = 0.995 , σ ( z 3 ) : = [ 1 + exp ( z 3 ) ] − 1 = 0.998 ℵ ( 0.5 ) : = v 1 ⋅ σ ( 0 .5 ⋅ w 1 + u 1 ) 2 + v 2 ⋅ σ ( 0 .5 ⋅ w 2 + u 2 ) 2 + v 3 ⋅ σ ( 0 .5 ⋅ w 3 + u 3 ) 2 = 3.199 ℵ : = v 1 ⋅ σ ( z 1 ) 2 + v 2 ⋅ σ ( z 2 ) 2 + v 3 ⋅ σ ( z 3 ) 2 = 3.01 y p ( x ) : = − 0.5 ⋅ ℵ ( 0.5 ) + x ⋅ ℵ = 1.41 , y d ( x ) : = ( 8 ⋅ x 3 − 1 ) ( x + 4 ⋅ x 4 ) − 1 = 1.4 , E : = 0.5 ⋅ ( y d ( x ) − y p ( x ) ) 2 = 5 × 10 − 5
| Input data (X) | 0.1 | 0.2 | 0.3 | 0.4 | 0.5 | 0.6 | 0.7 | 0.8 | 0.9 | 1 | 1.1 |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Y Exact | −9.881 | −4.535 | −2.359 | −0.971 | 0 | 0.651 | 1.050 | 1.269 | 1.371 | 1.4 | 1.3869 |
| Y Pred | −1.245 | −0.953 | −0.66 | −0.368 | −0.075 | 0.218 | 0.51 | 0.803 | 1.095 | 1.388 | 1.68 |
The plots of the exact and predicted values in
A novel Neural Network approach was developed recently by Okereke, for solving first and second order linear ordinary differential equations. In this article, the procedure is now extended in this article to investigate neural network solutions to nonlinear differential equations of Ricatti-type. Specifically, we employ a feed-forward Multilayer Perceptron Neural Network (MLPNN), but avoid the standard back-propagation algorithm for updating the intrinsic weights. This greatly reduces the computational complexity of the given problem. For desired accuracy our objective is to minimize an error, which is a function of the network parameters i.e., the weights and biases. Once the weights of the neural network are obtained by our systematic procedure, we need not adjust all the parameters in the network, as postulated by many researchers before us, in order to achieve convergence. We only need to fine-tune our biases which are fixed to lie in a certain given interval, and convergence to a solution with an acceptable minimum error is achieved. The first example ODE of Ricatti type to which the procedure is applied gave us perfect agreement with the exact solution. The second example however provided us with only an acceptable approximation to the exact solution. This has demonstrated quite clearly the function approximation capabilities of ANN in the solution of nonlinear differential equations of Ricatti type. The above method still requires some refinement so that it can be generalized to solve any type of nonlinear differential equation including partial differential equations.
The authors declare no conflicts of interest regarding the publication of this paper.
Okereke, R.N. and Maliki, O.S. (2021) Solving Riccati-Type Nonlinear Differential Equations with Novel Artificial Neural Networks. Applied Mathematics, 12, 919-930. https://doi.org/10.4236/am.2021.1210060