In this paper, an efficient computational algorithm is proposed to solve the nonlinear optimal control problem. In our approach, the linear quadratic optimal control model, which is adding the adjusted parameters into the model used, is employed. The aim of applying this model is to take into account the differences between the real plant and the model used during the calculation procedure. In doing so, an expanded optimal control problem is introduced such that system optimization and parameter estimation are mutually interactive. Accordingly, the optimality conditions are derived after the Hamiltonian function is defined. Specifically, the modified model-based optimal control problem is resulted. Here, the conjugate gradient approach is used to solve the modified model-based optimal control problem, where the optimal solution of the model used is calculated repeatedly, in turn, to update the adjusted parameters on each iteration step. When the convergence is achieved, the iterative solution approaches to the correct solution of the original optimal control problem, in spite of model-reality differences. For illustration, an economic growth problem is solved by using the algorithm proposed. The results obtained demonstrate the efficiency of the algorithm proposed. In conclusion, the applicability of the algorithm proposed is highly recommended.
Recently, the integrated optimal control and parameter estimation (IOCPE) algorithm has been proposed [
In fact, the basic idea for ISOPE, DISOPE and IOCPE algorithms is the principle of model-reality differences [
By virtue of the evolution of these algorithms, the feedback optimal control law is provided in solving the nonlinear optimal control problems, and their effectiveness has been well-confirmed. Nevertheless, the applicability of the open-loop optimal control law in these algorithms shall be investigated such that the popularity of these algorithms could be promoted. This is because of the open-loop optimal control sequences could be generated by taking the advantage of the power of the state-of-the-art nonlinear programming (NLP) solver. Thus, as an efficient optimization technique, the conjugate gradient method [
Hence, the application of the conjugate gradient method is discussed in this paper for solving the nonlinear optimal control problem, where the model-reality differences are considered. Apparently, the model-based optimal control problem, which is simplified from the nonlinear optimal control problem, is constructed. Follow from this, the Hamiltonian function is defined and the augmented cost function is obtained. Then, the set of the necessary conditions for optimality is derived. Consequently, the modified model-based optimal control problem is converted to be a nonlinear optimization problem. By applying the conjugate gradient approach, the nonlinear optimization problem is solved and the optimal control sequences are generated. With this open-loop control law, the dynamical system is optimized and the cost function is evaluated. For illustration, optimal control of an economic growth problem [
The structure of the paper is organized as follows. In Section 2, the problem statement is described briefly, where the simplified model from the nonlinear optimal control problem is discussed. In Section 3, system optimization with parameter estimation is further discussed. The use of the conjugate gradient approach in solving the model-based optimal control problem is presented and the calculation procedure is summarized as an iterative algorithm. In Section 4, an economic growth problem is solved and the results are obtained. Finally, the concluding remarks are made.
Consider a general discrete-time optimal control problem, given by
min u ( k ) J 0 ( u ) = φ ( x ( N ) , N ) + ∑ k = 0 N − 1 L ( x ( k ) , u ( k ) , k ) subjectto x ( k + 1 ) = f ( x ( k ) , u ( k ) , k ) , x ( 0 ) = x 0 (1)
where u ( k ) ∈ ℜ m , k = 0 , 1 , ⋯ , N − 1 and x ( k ) ∈ ℜ n , k = 0 , 1 , ⋯ , N are, respectively, the control sequences and the state sequences. Here, f : ℜ n × ℜ m × ℜ → ℜ n represents the real plant, L : ℜ n × ℜ m × ℜ → ℜ is the cost under summation and φ : ℜ n × ℜ → ℜ is the terminal cost, whereas J 0 is the scalar cost function and x 0 is the known initial state vector. It is assumed that all functions in Equation (1) are continuously differentiable with respect to their respective arguments.
This problem is regarded as the real optimal control problem, and is referred to as Problem (P). Note that the structure of Problem (P) is complex and nonlinear, solving Problem (P) requires the efficient computation techniques. On this point of view, the simplified model of Problem (P) is probably suggested to be solved in order to approximate the true optimal solution of Problem (P). Therefore, let us define this simplified model-based optimal control problem as follows:
min u ( k ) J 1 ( u ) = 1 2 x ( N ) T S ( N ) x ( N ) + γ ( N ) + ∑ k = 0 N − 1 1 2 ( x ( k ) T Q x ( k ) + u ( k ) T R u ( k ) ) + γ ( k ) subjectto x ( k + 1 ) = A x ( k ) + B u ( k ) + α ( k ) , x ( 0 ) = x 0 (2)
where γ ( k ) , k = 0 , 1 , ⋯ , N and α ( k ) , k = 0 , 1 , ⋯ , N − 1 are introduced as the adjusted parameters, whereas A is an n × n transition matrix and B is an n × m control coefficient matrix. Besides, S ( N ) and Q are n × n positive semi-definite matrices, and R is a m × m positive definite matrix. Here, J 1 is the scalar cost function.
This problem is referred to as Problem (M).
Notice that, due to the different structures and parameters, only solving Problem (M), without the adjusted parameters, would not obtain the optimal solution of Problem (P). However, by adding the adjusted parameters into Problem (M), the differences between the real plant and the model used can be calculated. In such a way, solving Problem (M) iteratively could give the correct optimal solution of Problem (P), in spite of model-reality differences.
Now, introduce an expanded optimal control problem, which is referred to as Problem (E), given by
min u ( k ) J 2 ( u ) = 1 2 x ( N ) T S ( N ) x ( N ) + γ ( N ) + ∑ k = 0 N − 1 1 2 ( x ( k ) T Q x ( k ) + u ( k ) T R u ( k ) ) + γ ( k ) + 1 2 r 1 ‖ u ( k ) − v ( k ) ‖ 2 + 1 2 r 2 ‖ x ( k ) − z ( k ) ‖ 2 subjectto x ( k + 1 ) = A x ( k ) + B u ( k ) + α ( k ) , x ( 0 ) = x 0 1 2 z ( N ) T S ( N ) z ( N ) + γ ( N ) = φ ( z ( N ) , N )
1 2 ( z ( k ) T Q z ( k ) + v ( k ) T R v ( k ) ) + γ ( k ) = L ( z ( k ) , v ( k ) , k ) A z ( k ) + B v ( k ) + α ( k ) = f ( z ( k ) , v ( k ) , k ) v ( k ) = u ( k ) z ( k ) = x ( k ) (3)
where v ( k ) ∈ ℜ m , k = 0 , 1 , ⋯ , N − 1 and z ( k ) ∈ ℜ n , k = 0 , 1 , ⋯ , N are introduced to separate the sequences of control and state in the optimization problem from the respective signals in the parameter estimation problem, and ‖ ⋅ ‖ de
notes the usual Euclidean norm. The term 1 2 r 1 ‖ u ( k ) − v ( k ) ‖ 2 and 1 2 r 2 ‖ x ( k ) − z ( k ) ‖ 2 with r 1 , r 2 ∈ ℜ are introduced to improve the convexity and
to facilitate the convergence of the resulting iterative algorithm. Here, it is classified that the algorithm is designed such that the constraints v ( k ) = u ( k ) and z ( k ) = x ( k ) are satisfied upon termination of the iterations, assuming that convergence is achieved. Moreover, the state constraint z ( k ) and the control constraint v ( k ) are used for the computation of the parameter estimation and matching scheme, while the corresponding state constraint x ( k ) and control constraint u ( k ) are reserved for optimizing the model-based optimal control problem. By virtue of this, system optimization and parameter estimation are mutually integrated.
Define the Hamiltonian function for Problem (E) by
H 2 ( k ) = 1 2 ( x ( k ) T Q x ( k ) + u ( k ) T R u ( k ) ) + γ ( k ) + 1 2 r 1 ‖ u ( k ) − v ( k ) ‖ 2 + 1 2 r 2 ‖ x ( k ) − z ( k ) ‖ 2 + p ( k + 1 ) T ( A x ( k ) + B u ( k ) + α ( k ) ) − λ ( k ) T u ( k ) − β ( k ) T x ( k ) (4)
where λ ( k ) ∈ ℜ m , k = 0 , 1 , ⋯ , N − 1 , β ( k ) ∈ ℜ n , k = 0 , 1 , ⋯ , N and p ( k ) ∈ ℜ n , k = 0 , 1 , ⋯ , N are modifiers. Then, the augmented cost function becomes
J ′ 2 ( u ) = 1 2 x ( N ) T S ( N ) x ( N ) + γ ( N ) + p ( 0 ) T x ( 0 ) − p ( N ) T x ( N ) + ξ ( N ) ( φ ( z ( N ) , N ) − 1 2 z ( N ) T S ( N ) z ( N ) − γ ( N ) ) + Γ T ( x ( N ) − z ( N ) ) + ∑ k = 0 N − 1 H 2 ( k ) − p ( k ) T x ( k ) + λ ( k ) T v ( k ) + β ( k ) T z ( k ) + ξ ( k ) ( L ( z ( k ) , v ( k ) , k ) − 1 2 ( z ( k ) T Q z ( k ) + v ( k ) T R v ( k ) ) − γ ( k ) ) + μ ( k ) T ( f ( z ( k ) , v ( k ) , k ) − A z ( k ) − B v ( k ) − α ( k ) ) (5)
where p ( k ) , ξ ( k ) , λ ( k ) , β ( k ) , μ ( k ) and Γ are the appropriate multipliers to be determined later.
Applying the calculus of variation [
1) Stationary condition:
R u ( k ) + B T p ( k + 1 ) − λ ( k ) + r 1 ( u ( k ) − v ( k ) ) = 0 (6a)
2) Co-state equation:
p ( k ) = Q x ( k ) + A T p ( k + 1 ) − β ( k ) + r 2 ( x ( k ) − z ( k ) ) (6b)
3) State equation:
x ( k + 1 ) = A x ( k ) + B u ( k ) + α ( k ) (6c)
4) Boundary conditions:
p ( N ) = S ( N ) x ( N ) + Γ and x ( 0 ) = x 0 (6d)
5) Adjusted parameter equations:
φ ( z ( N ) , N ) = 1 2 z ( N ) T S ( N ) z ( N ) + γ ( N ) (7a)
L ( z ( k ) , v ( k ) , k ) = 1 2 ( z ( k ) T Q z ( k ) + v ( k ) T R v ( k ) ) + γ ( k ) (7b)
f ( z ( k ) , v ( k ) , k ) = A z ( k ) + B v ( k ) + α ( k ) (7c)
6) Modifier equations:
Γ = ∇ z ( N ) φ − S ( N ) z ( N ) (8a)
λ ( k ) = − ( ∇ v ( k ) L − R v ( k ) ) − ( ∂ f ∂ v ( k ) − B ) T p ^ ( k + 1 ) (8b)
β ( k ) = − ( ∇ z ( k ) L − Q z ( k ) ) − ( ∂ f ∂ z ( k ) − A ) T p ^ ( k + 1 ) (8c)
with ξ ( k ) = 1 and μ ( k ) = p ^ ( k + 1 ) .
7) Separable variables:
v ( k ) = u ( k ) , z ( k ) = x ( k ) , p ^ ( k ) = p ( k ) . (9)
Notice that the parameter estimation problem is defined by Equation (7) and the computation of multipliers is given by Equation (8). Indeed, the necessary conditions, which are defined by Equations (6a) to (6d), are the optimality for the modified model-based optimal control problem.
The modified model-based optimal control problem, which is referred to as Problem (MM), is given by
min u ( k ) J 3 ( u ) = 1 2 x ( N ) T S ( N ) x ( N ) + Γ T x ( N ) + γ ( N ) + ∑ k = 0 N − 1 1 2 ( x ( k ) T Q x ( k ) + u ( k ) T R u ( k ) ) + γ ( k ) + 1 2 r 1 ‖ u ( k ) − v ( k ) ‖ 2 + 1 2 r 2 ‖ x ( k ) − z ( k ) ‖ 2 − λ ( k ) T u ( k ) − β ( k ) T x ( k ) subjectto x ( k + 1 ) = A x ( k ) + B u ( k ) + α ( k ) , x ( 0 ) = x 0 (10)
with the specified α ( k ) , γ ( k ) , λ ( k ) , β ( k ) , Γ , v ( k ) and z ( k ) , where the boundary conditions are given by x 0 and p ( N ) with the specified multiplier Γ .
For simplicity, define Problem (MM) as an equivalent nonlinear optimization problem with the initial control u ( 0 ) = u ( k ) 0 , given by
min u ( k ) J 3 ( u ) subject to u ∈ ℜ m for k = 0 , 1 , ⋯ , N − 1 (11)
where the admissible control variable u is set to be
u = [ ( u ( 0 ) ) T , ( u ( 1 ) ) T , ⋯ , ( u ( N − 1 ) ) T ] .
Let this problem as Problem (N). To proceed, it is noticed that solving Problem (N) could be done once the state Equation (6c) is solved forward and the costate Equation (6b) is solved backward with the corresponding control sequences u. In addition, the gradient function for the objective function J 3 ( u ) is evaluated from
∇ J 3 ( u ) = ∇ u ( k ) H (12)
which can be calculated from the Hamiltonian function (4) and the stationary condition (6a) once the necessary conditions for optimality, given by Equations (6) - (9), are satisfied.
Suppose the gradient function (12) is represented as
g ( u ) = ∇ u ( k ) H . (13)
Then, for arbitrary initial control u ( 0 ) , the initial gradient and the initial direction are, respectively, given by
g ( 0 ) = g ( u ( 0 ) ) and d ( 0 ) = − g ( 0 ) . (14)
By using the line search equation [
u ( i + 1 ) = u ( i ) + a i ⋅ d ( i ) (15)
where a i ∈ ℜ is determined from the one-dimensional search, that is,
a i = arg min a ≥ 0 J 3 ( u ( i ) + a ⋅ d ( i ) ) . (16)
Later, the gradient and the direction are updated as follow:
g ( i + 1 ) = g ( u ( i + 1 ) ) (17)
d ( i + 1 ) = − g ( i + 1 ) + b i ⋅ d ( i ) (18)
with the coefficient
b i = g ( i + 1 ) T g ( i + 1 ) g ( i ) T g ( i ) (19)
where i = 0 , 1 , 2 , ⋯ represents the iteration numbers.
Thus, we present the result on the obtaining optimal control law discussed above as a proposition, given below:
Proposition 1. Consider Problem (N). The control sequences u ( i ) , which is defined in Equation (15) and is represented by
u ( i ) = [ ( u ( 0 ) ) T , ( u ( 1 ) ) T , ⋯ , ( u ( N − 1 ) ) T ] ,
is generated through a set of the direction vectors d ( i ) whose components are linearly independent. Also, the direction d ( i ) is conjugacy.
Proof: Refer [
Here, the conjugate gradient algorithm for obtaining the optimal control law is summarized below:
Algorithm 1: Conjugate gradient algorithm
Data Choose the arbitrary initial control u ( 0 ) . Compute the initial gradient g ( 0 ) and the initial direction d ( 0 ) from Equation (14). Set i = 0.
Step 1 Solve the state Equation (6c) forward in time from k = 0 to k = N with the initial condition (6d) to obtain x ( k ) i , k = 0 , 1 , ⋯ , N .
Step 2 Solve the costate Equation (6b) backward in time from k = N to k = 0 with the boundary condition (6d), where p ( k ) i is the solution obtained.
Step 3 Calculate the value of the cost functional J 3 ( u ( i ) ) from Equation (10).
Step 4 Determine the step size a i from Equation (16).
Step 5 Update the control u ( i + 1 ) from Equation (15).
Step 6 Update the gradient g ( i + 1 ) from Equation (17). If the gradient g ( i + 1 ) = 0 , stop, else go to Step 7.
Step 7 Compute the coefficient b i from Equation (19).
Step 8 Update the direction d ( i + 1 ) from Equation (18). Set i = i + 1 , go to Step 1.
Remarks:
1) The initial control u ( 0 ) can be any valued-vectors, including the zero vector.
2) The gradient function g ( u ) for Problem (N) defined by Equation (11) is calculated from the stationary condition (6a). This is the turning point of using the conjugate gradient algorithm for solving Problem (M) defined by Equation (2) and Problem (MM) defined by Equation (10).
3) The optimal control sequences generated by the line search equation in Equation (15) is known as the open-loop control law.
4) The necessary conditions (6b) and (6c) shall be satisfied in solving Problem (N) defined by Equation (11).
Accordingly, from the discussion above, a summary of the calculation procedure for the integrated system optimization and parameter estimation is made as follows:
Algorithm 2: Iterative procedure
Data A , B , Q , R , S ( N ) , N , x 0 , r 1 , r 2 , k v , k z , k p , f , L . Note that A and B could be determined based on the linearization of f at x 0 or from the linear terms of f .
Step 0 Compute a nominal solution. Assume that α ( k ) = 0 , k = 0 , 1 , ⋯ , N − 1 and r 1 = r 2 = 0 . Solve Problem (M) defined by Equation (2) to obtain u ( k ) 0 , k = 0 , 1 , ⋯ , N − 1 and x ( k ) 0 , p ( k ) 0 , k = 0 , 1 , ⋯ , N . Then, with α ( k ) = 0 , k = 0 , 1 , ⋯ , N − 1 and using r 1 , r 2 from the data. Set i = 0 , v ( k ) 0 = u ( k ) 0 , z ( k ) 0 = x ( k ) 0 and p ^ ( k ) 0 = p ( k ) 0 .
Step 1 Compute the parameters γ ( k ) i , k = 0 , 1 , ⋯ , N and α ( k ) i , k = 0 , 1 , ⋯ , N − 1 from Equation (7). This is called the parameter estimation step.
Step 2 Compute the modifiers Γ i , λ ( k ) i and β ( k ) i , k = 0 , 1 , ⋯ , N − 1 from Equation (8). Notice that this step requires taking the derivatives of f and L with respect to v ( k ) i and z ( k ) i .
Step 3 With γ ( k ) i , α ( k ) i , Γ i , λ ( k ) i , β ( k ) i , v ( k ) i and z ( k ) i , solve Problem (N) by using Algorithm 1. This is called the system optimization step.
a) Use Equation (15) to obtain the new control u ( k ) i , k = 0 , 1 , ⋯ , N − 1 .
b) Use Equation (6c) to obtain the new state x ( k ) i , k = 0 , 1 , ⋯ , N .
c) Use Equation (6b) to obtain the new costate p ( k ) i , k = 0 , 1 , ⋯ , N .
Step 4 Test the convergence and update the optimal solution of Problem (P). In order to provide a mechanism for regulating convergence, a simple relaxation method is employed:
v ( k ) i + 1 = v ( k ) i + k v ( u ( k ) i − v ( k ) i ) (20a)
z ( k ) i + 1 = z ( k ) i + k z ( x ( k ) i − z ( k ) i ) (20b)
p ^ ( k ) i + 1 = p ^ ( k ) i + k p ( p ( k ) i − p ^ ( k ) i ) (20c)
where k v , k z , k p ∈ ( 0 , 1 ] are scalar gains. If v ( k ) i + 1 = v ( k ) i , k = 0 , 1 , ⋯ , N − 1 and z ( k ) i + 1 = z ( k ) i , k = 0 , 1 , ⋯ , N , within a given tolerance, stop; else set i = i + 1 , and repeat the procedure starting from Step 1.
Remarks:
1) The variable α ( k ) is zero in Step 0. The calculated value of α ( k ) changes from iteration to iteration during the calculation procedure.
2) The conjugate gradient algorithm is applied to generate the control sequences u ( k ) for Problem (M) and Problem (MM), respectively.
3) Problem (P) is not necessary to be linear or to have a quadratic cost function.
4) The conditions u ( k ) i + 1 = u ( k ) i and x ( k ) i + 1 = x ( k ) i are required to be satisfied for the converged optimal control sequence and the converged state sequence. The following averaged 2-norms are computed and then they are compared with a given tolerance to verify the convergence of u ( k ) and x ( k ) :
‖ u i + 1 − u i ‖ 2 = ( 1 N − 1 ∑ k = 0 N − 1 ‖ u ( k ) i + 1 − u ( k ) i ‖ ) 1 / 2 (21a)
‖ x i + 1 − x i ‖ 2 = ( 1 N ∑ k = 0 N ‖ x ( k ) i + 1 − x ( k ) i ‖ ) 1 / 2 (21b)
5) The convergence result on the conjugate gradient algorithm can be referred to [
Consider a basic economic growth model [
min u ( k ) J 0 ( u ) = ∑ k = 0 N − 1 ρ k L ( x ( k ) , u ( k ) , k ) subjectto x ( k + 1 ) = f ( x ( k ) , u ( k ) , k ) , x ( 0 ) = 5
where the payoff function and dynamics system are, respectively, defined by
L ( x ( k ) , u ( k ) , k ) = − ln ( Λ x ( k ) ν − u ( k ) ) and f ( x ( k ) , u ( k ) , k ) = u ( k ) .
Here, x is the capital stock, x is the control variable, ρ is the discount factor, whereas Λ x ν is a production function with constants Λ > 0 , 0 < ν < 1 . The difference between the output and the next period’s capital stock is the consumption.
Let us refer this problem as Problem (P). In literature, the exact solution of Problem (P) is known [
V ( x ) = C + D ln ( x ) ,
with
C = ln ( ( 1 − ν ⋅ ρ ) ⋅ Λ ) + D ⋅ ρ ⋅ ln ( ν ⋅ ρ ⋅ Λ ) 1 − ρ and D = ν 1 − ν ⋅ ρ .
The unique optimal equilibrium for Problem (P) is given by
x * = 1 ν ⋅ ρ ⋅ Λ ( ν − 1 ) .
By using the specified parameters Λ = 5 , ν = 0.34 and ρ = 0.95 , the optimal equilibrium is x * ≈ 2.0673 [
In the following, we introduce a simplified model-based optimal control model, which is derived from Problem (P) and is referred to as Problem (M), given below:
min u ( k ) J 1 ( u ) = ∑ k = 0 30 ( 1 2 ( 0.1 ⋅ x ( k ) 2 + 10 ⋅ u ( k ) 2 ) + γ ( k ) ) subjectto x ( k + 1 ) = x ( k ) + u ( k ) + α ( k ) , x ( 0 ) = 5.
Note that Problem (M) and Problem (P) are different from the structures and the parameters used.
After running the algorithm proposed within the tolerance (10−6), the result is shown in
The graphical results for this economic growth model illustrate the application of the algorithm proposed.
The use of the conjugate gradient approach in solving the nonlinear optimal control problem with model-reality differences was discussed in this paper. Essentially, the simplified model of the original optimal control problem, which is the linear optimal control problem by adding the adjusted parameters, is
| Iteration Number | Initial Cost | Final Cost | Elapsed Time (s) |
|---|---|---|---|
| 41 | 13.072 | 22.198 | 7.84008 |
formulated. In solving this model-based optimal control problem, the conjugate gradient approach is employed to generate the open-loop control sequences such that the optimal solution is obtained. Here, the stationary condition is used to be the gradient function in the conjugate gradient approach. On the other hand, due on the different structure of the problems, the differences between the real plant and the model used, which is measured by the adjusted parameters repeatedly, are taken into consideration during the iteration calculation procedure. At the convergence, the optimal solution of the model used approximates to the true optimal solution of the original optimal control problem, in spite of model-reality differences. For illustration, the application of the algorithm proposed was discussed for solving a basic economic growth model. The results obtained show the efficiency of the algorithm proposed. In conclusion, the applicability of the algorithm is highly recommended.
The authors would like to acknowledge the Universiti Tun Hussein Onn Malaysia (UTHM) and the Ministry of Higher Education (MOHE) for the financial support for this study under the research grant FRGS VOT. 1561.
The authors declare no conflicts of interest regarding the publication of this paper.
Kek, S.L., Leong, W.J., Sim, S.Y. and Teo, K.L. (2018) Application of Conjugate Gradient Approach for Nonlinear Optimal Control Problem with Model-Reality Differences. Applied Mathematics, 9, 940-953. https://doi.org/10.4236/am.2018.98064