The process of minimizing or maximizing a problem is called optimization or mathematical programming as it is termed in standard texts. Optimization is all about finding the best possible solution in a given set of circumstances. Most real life problems always appear as non-linear multivariate problems when expressed mathematically [1]. Sometimes, these problems may have some conditions which its decision variables must satisfy and in this case, we say the problem is constrained, otherwise the problem is unconstrained. Optimization involves choices informed by the study of the situations and parameters affecting those situations in order to either minimize cost/efforts or maximize benefits. According to Chong and Zak [2], optimization is central to any problem involving decision making, whether in engineering or in economics. The task of decision making entails choosing between various alternatives. This choice is governed by our desire to make the best decision. The measure of goodness of the alternatives is informed by an objective function or performance index. In this research article, we are concerned with solving some non-linear constrained optimization problems using artificial neural network approach.

While penalty-based neural network methods for constrained optimization have been established in the literature [3][4], the practical implementation of these methods often remains abstract, with limited step-by-step numerical demonstrations, particularly for inequality-constrained nonlinear problems. The novelty of the present work is threefold:

1) We provide a fully transparent, stepbystep derivation from the penalty function to the explicit ODE system for two concrete test problems;

2) We demonstrate, using MathCAD14, how the choice of penalty parameters k i and learning rates μ j critically affects convergence to the true constrained optimum—a pragmatic issue often underreported; and

3) We present comparative graphical evidence that, for the problems considered, the neural ODE approach converges to the analytic solution from multiple initial conditions. Thus, unlike prior works that focus on theoretical architecture or general algorithms, this paper offers a reproducible numerical recipe and a candid discussion of parameter tuning.

A neural network is an inter-connection of processing elements, units or nodes, whose functionality is similar to that of the human neurons [5][6]. The processing ability of the network is stored in the connection strengths, simply called weights, which can be obtained by a process of adaptation to, a set of training patterns. Neural network methods can solve both linear and non-linear optimization problems. This is possible due to the function approximation property [7] of feed forward neural networks which converts the optimization problem into an ordinary differential equation. This in turn can be solved by conventional methods.

ANNs are constructed with layers of units [8], which are termed multilayer ANNs. A layer of units in an ANN is made up of units that perform similar tasks. First layer of a multilayer ANN consists of input units. These units perform similar tasks as independent variables in statistical literature. Last layer contains output units known as dependent or response variables in statistical nomenclature. Between input and output units in the model are other units called hidden units and constitute hidden layers. There are two functions that govern the behaviour of a unit in a particular layer, which normally are the same for all units within the whole ANN, i.e.

the input function andthe output/activation function

Input into a node is a weighted sum of output from nodes connected to it.

A neuron receives a set of n inputs S = { x j | j = 1 , 2 , ⋯ , n } . In Figure 1, each input is weighted before reaching the main body of a neuron N i by connection strength or weight factor w i j for j = 1 , 2 , ⋯ , n . In addition, it has a bias b or w 0 , a threshold value θ k , which has to be reached or exceeded for the neuron to produce an output signal. A function φ ( s ) acts on the produced weighted signal. This function is called an activationfunction. Mathematically, the output of the i^th neuron N i is given by;

Figure 1. Mathematical model of artificial neural network.

And the neuron’s firing condition is,

Figure 1 shows detailed computational steps of the working principle of an artificial neuron (AN) in a neural network. Now the input signal for the i^th neuron N i is,

Artificial neural network can make a nonlinear mapping from the inputs to the outputs of the corresponding system of neurons, which is suitable for analyzing the problem defined by initial/boundary value problems that have no analytical solutions or which cannot be easily computed. One of the applications of the multilayer feed forward neural network is the global approximation of real valued multivariable function in a closed analytic form. Namely such neural networks are universal approximators. It is discovered in the literature that multilayer feed forward neural networks with one hidden layer using arbitrary squashing functions are capable of approximating any Borel measurable function from one finite dimensional space to another with any desired degree of accuracy. This is made clear in the following theorem.

The universal approximation theorem for MLP was proved by Cybenko [7] and Hornik etal. [8] in 1989. Let I n represent an n-dimensional unit cube containing all possible input samples x = ( x 1 , x 2 , ⋯ , x n ) with x i ∈ [ 0 , 1 ] , i = 1 , 2 , ⋯ , n . Let C ( I n ) be the space of continuous functions on I n , given a continuous sigmoid function φ ( ⋅ ) , then the universal approximation theorem states that the finite sums of the form

are dense in C ( I n ) . This simply means that given any function f ∈ C ( I n ) and ε > 0 , there is a sum y ( x , w ) of the above form that satisfies

The neural networks follow a different paradigm for computing. The conventional methods to solution are good for fast arithmetic and does what programmer programs. But the conventional methods to solution are not so good for interacting with noisy data or data from the environment massive parallelism, fault tolerance and adopting to circumstances.

The neural network systems help where we cannot formulate an algorithmic solution.

Neutral networks are a form of multiprocessor computer system with.

Learning ability Fault tolerance Simple processing elements A high degree of interconnectionA simple scalar messages andAdaptive interaction between elementsLow energy consumption.

Most importantly neutral network converts the non-linear optimization problem together with the constraints to a set of ordinary differential equation which can easily be solved either analytically or numerically.

Many constraint optimization problems have been solved using Khun-Trucker and Lagrange multiplier methods [9]. Most of these methods still give concern because of their vigorous steps and assumptions. To solve constrained non-linear optimization problems using artificial neural networks, we employ penalty functions that transform the constrained problem into a single unconstrained problem. The constraints are placed into the objective function via a penalty parameter in such a way that it penalizes any violation of the constraints. The penalty function constructed is actually an energy function for the neural network. Assuming differentiability, a local minimum of the penalty function is found by using a dynamic gradient scheme which provides a system of differential equations for the input neurons corresponding to the variables of the given optimization problem.

We now consider a brief overview of artificial neural network structures employed in the solution of constrained optimization problems with inequality constraints. The general form of the problem to be solved is

subject to g i ( x ) ≥ 0 , i = 1 , 2 , ⋯ , m

where x = ( x 1 , x 2 , ⋯ , x n ) T . In Equation (2) f ( x ) is the objective function which has to be minimized and g i ( x ) ,   i = 1 , 2 , ⋯ , m are inequality constraints. It should be noted that any equality constraint h i ( x ) , could be transformed into two inequality constraints h i ( x ) ≥ 0 , − h i ( x ) ≥ 0 , i.e. the problem (5) is sufficiently general. All functions involved are assumed to be continuously differentiable. We have the following important theorem.

Theorem1. Let ( x k ) be a sequence generated by the penalty method with { s k } , k = 1 , 2 , ⋯ , being a nonnegative and strictly increasing sequence tending to infinity. Then, any limit point of the sequence ( x k ) is a solution to the problem. The proof can be found in Luenberger [10].

The penalty function method consists of transforming the constrained optimization problem into an unconstrained one based on a penalty function defined as follows:

where the penalty term P ( g i ( x ) ) should satisfy

The unconstrained problems are solved by adding a term, called a penalty function to the objective function that consists of a penalty parameter multiplied by a measure of violation of the constraint. In the neural network literature the penalty function E ( x , k ) is often considered as an energy function for the corresponding neural network [11]. k i = k , i = 1 , 2 , ⋯ , n . Here, two types of penalty terms are used

Based on the penalty terms (8), the energy function obtained is

Assuming that g i ( x ) ,   i = 1 , 2 , ⋯ , m have continuous first derivatives, the same is true for P ( g i ( x ) ) as defined by (8), therefore E ( x , k ) in (10) is continuously differentiable. The gradient of P ( g i ( x ) ) in (8) is:

By using the gradient strategy for minimizing E ( x , k ) , the following system of ordinary differential equations is obtained

In extended form, the above can be written

where μ j > 0 j = 1 , 2 , ⋯ , n , represent the learning rate parameters, k i > 0 , i = 1 , 2 , ⋯ , m , and t is the neural network time, with

and, obviously,

The functional scheme for simulating the above equations could be considered as a Neural Network, where the integrators represent the neurons (basic units) and functional nonlinear generators build up the connections between them. This approach, described by Cichocki etal. [12], essentially replaces the constrained problem (5) by an unconstrained minimization of the differentiable penalty function (6). However, theoretical results on the penalty function method, show that

equivalence of the problems (5) and min x { E ( x , k ) } is only obtained in the limit, as min i = 1 , ⋯ , m { k i } → ∞ .

The quality of the approximation improves provided larger values are chosen for the k i ’s, however, the true solution of (5) will not be obtained unless

Let us now consider the penalty terms (7). Using these in (10) gives the following penalty function:

Due to the non-differentiability of the penalty term (7), this penalty function is non-differentiable, even though the functions g i ( x ) are assumed differentiable. When the penalty terms (7) are used, the corresponding system of ordinary differential equations is:

This is a simpler neural network than the previous one. However because of the non-smooth nature of the penalty function, the first derivative discontinuities could slow up the solution speed [13].

Given: Min f ( x ) s.t g i ( x ) ≥ 0 ,   i = 1 , 2 , ⋯ , m

1) Form the penalty function as given by Equation (10):

2) Derive the ODE system via gradient descent:

where S i = 1 if g i ( x ) ≤ 0 (active/violated), else S i = 0 .

3) Choose penalty parameters k i > 0 (start moderate, e.g., 2 - 20) and learning rates μ j > 0 (e.g., 10 - 100).

4) Set initial conditions x ( 0 ) —preferably inside the feasible region or slightly outside.

5) Solve the ODE system numerically (e.g., Runge-Kutta adaptive) over an interval t ∈ [ 0 , T ] until d x / d t ≈ 0 .

6) Check feasibility: verify g i ( x ∗ ) ≥ 0 within tolerance. Check optimality: compare with known analytic solution or verify KKT conditions.

7) If constraints are violated, increase k i and/or adjust μ j and resolve.

Remark:

We now address the issue of the choice of parameters and tuning rule.

In all simulations, learning rates μ j were set equal and chosen to ensure stable integration without oscillation (typical range 10 - 100). Penalty parameters k i were initially set to 2. If the steady-state solution violated any constraint, k i was increased multiplicatively (e.g., from 2 to 20) until all constraints were satisfied within a tolerance of 10⁻³. Initial conditions were selected to test convergence from different basins; when convergence led to infeasibility, we re-initialized closer to the expected feasible region (as in Problem 2, moving from (1, 0, 1) to (15, 10, 5)). The integration interval [ 0 , T ] was chosen as T = 2 because all trajectories reached steady state by t ≈ 1 . No automatic stopping criterion was used; steady state was identified visually from the graphical profiles.

The penalty method transforms the constrained problem into an unconstrained one, but equivalence is only attained in the limit as min i { k i } → ∞ . In practice, finite k i yield approximate solutions. Convergence depends sensitively on the choice of penalty parameters, learning rates, and initial conditions, as demonstrated in Problem 2. Poor choices can lead to infeasible steady states or slow convergence. Additionally, the claim that the neural network method is always easier and faster than classical approaches (e.g., Lagrange multiplier or sequential quadratic programming) is not supported by direct computational time comparisons here. For small-scale problems, classical methods may be equally efficient and offer exact constraint satisfaction without parameter tuning. Therefore, the advantages of the neural ODE method are most evident when an analytic gradient is available, the ODE system can be solved with standard integrators, and the user is willing to tune penalty parameters for feasibility.

In this section, we present the results for the neural network system considered. We present the simulation of the ODE models of the Neural Network obtained from solving our various problems. The graphical profile of the solution is also shown. We will then go on to discuss our results and then compare with the analytic solution of the constraint optimization problem. We first consider the following simple example in other to test the network.

Problem1

Subject to

The solution space for problem of Equation (14) is defined by the functions g 1 ( x ) and g 2 ( x ) depicted below in Figure 2.

Figure 2. Feasible region of problem (17) with solution x ∗ = ( x 1 , x 2 ) = ( 0 , 2 ) .

Following the algorithm of the penalty function term (16) associated with the Equation (2), we first develop the differential equation for the neurons x 1 , x 2 whose solution we expect will converge to the solution x ∗ = ( x 1 , x 2 ) = ( 0 , 2 ) . We then have;

Here S 1 = S 2 = 1 , since our constraints is not strictly greater than zero. Similarly,

Thus the neural network for the given optimization problem is

We employed MathCAD14 [14] for stimulation of the ODE model of the neural network. We started by choosing an initial condition;

x ( 0 ) = ( x 1 ( 0 ) , x 2 ( 0 ) ) = ( 1 , − 1 ) and x ( 0 ) = ( x 1 ( 0 ) , x 2 ( 0 ) ) = ( − 1 , 1 ) . Furthermore we employ the following parameters k 1 = 2 , k 2 = 2 , μ 1 = 10 , μ 2 = 10 . Solving the ODEs, we obtain the following graphical profiles for their solutions.

InFigure 3(a), we can see that x 1 ( t ) starting from its initial point of 1, converges to 0, while x 2 ( t ) started from its initial point of −1 converges to 2 which is the exact analytic solution to the optimization problem.

Figure 3. (a) x 1 ( t ) = e − 40 t , x 2 ( t ) = 2 − 3 e − 20 t ; (b) x 1 ( t ) = − e − 40 t , x 2 ( t ) = 2 − e − 20 t .

Similarly inFigure 3(b) with different initial conditions the system converges again to the exact solution.

Next, we consider higher dimensional non-linear constrained optimization problems.

Problem2

Subject to

Here m = 3 , j = 1 , 2 , 3 .

The neural network system of equations for the given optimization problem is

To solve the ODEs we first choose the following values k 1 = 2 , k 2 = 2 , k 3 = 2 , μ 1 = 10 , μ 2 = 10 , μ 3 = 10 . Also we maintain S 1 = S 2 = S 3 = 1 .

The above system is solved using MathCAD 14. The graphical profiles are given below.

Remark

1) In Figure 4(a), from the initial point of 1, the neural network rises sharply above 30 and then converged from t = 0 to a steady state solution of y₀ = x₁ = 37.526.

2) In Figure 4(b), from the initial point of 0, the neural network rises sharply above 40 and ten converged from t = 0 to a steady state solution of y₁ = x₂ = 44.286.

3) In Figure 4(c) from the initial point of 1, the neural network rises sharply above 50 and ten converged from t = 0 to a steady state solution of y₂ = x₃ = 54.286.

Figure 4. Graphical profile of the solutions to Problem 2.

From the values of our y, we can see that y₀ = x₁ = 37.526 did not satisfy the first constraint of the problem 2. We then adjusted our parameters upwards to have values; μ = 10, k₁ = 20, k₂ = 20, k₃ = 20. This in turn gave us a new system of neural network ODE, viz;

The steady state solution in this case is x₁ = 48.515, x₂ = 52.295 and x₃ = 62.295. We can see that at y₀ = x₁ = 48.515, did not still satisfy the first constraint of the problem 2. we then review the initial condition from (1, 0, 1) to (15, 10, 5) with the same parameters of μ = 10, k₁ = 20, k₂ = 20, k₃ = 20. The set of neural network ODE is the same as before, i.e. Equations (24), but with initial conditions (15, 10, 5). The steady state solutions are as follows:

From Figure 5, the steady state solutions are now x₁ = 50.243, x₂ = 52.295 and x₃ = 62.295. In this case we can see that all the values have converged to the optimal solution of the equation and satisfying all the constraints of the problem.

Figure 5.Graphical profile of the solutions to Problem 2 with adjusted parameters.

Problem3(Equalityconstraint)

Consider the equality constraint minimization problem

Subject to h ( x ) = x 1 + x 2 − 1 = 0

According to section 8, the equivalent inequality-constrained problem becomes:

Subject to { g 1 ( x ) = x 1 + x 2 − 1 ≥ 0 g 2 ( x ) = − x 1 − x 2 + 1 ≥ 0

Following the differentiable penalty form given by Equation (10), we have

here, g 1 ( x ) = x 1 + x 2 − 1 ≥ 0 and g 2 ( x ) = − x 1 − x 2 + 1 ≥ 0 .

Since both constraints must be nonnegative, the penalty activates only when either becomes negative. The resulting ODE system can then be derived using Equation (12) with m = 2 , S 1 and S 2 defined by Equation (13).

In this paper, we have been able to demonstrate how the neural network approach works in solving both equality and inequality non-linear programming problems as well as how to generate the system of equations that must correspond to the given number of variables. With the knowledge and understanding, we applied them in solving some constrained non-linear optimization problems. Using MathCAD, we were able to solve the system of equations developed by using the neural network gradient scheme. We presented our results graphically and compared it with the analytic solution. Our steps and results have shown that, for the problems tested, the neural network approach is more straightforward to implement and avoids some of the iterative assumptions of classical methods.