The conjugate gradient method (CG) and Quasi-Newton method are two major popular iterative methods for solving smooth unconstrained optimization problems, and Broyden-Fletcher-Goldfarb-Shanno (BFGS) method is one of the most efficient quasi-Newton methods for solving small and medium sized unconstrained optimization problems [1] [2] [3] [4]. The iterative method is computed by

x k + 1 = x k + α k d k , (1)

where α k is a step size and d k is a search direction. For continuously differentiable function h : R n → R , the minimization problem:

min x ∈ R n h ( x ) (2)

has been well studied for several decades. Conjugate gradient method is among the preferable methods for solving problem (2) with search direction d k given by

d k = ( − ∇ h k + β k d k − 1           if   k   ≥ 1 , − ∇ h k                     if   k   = 0 , (3)

where ∇ h k is the gradient of an objective function h ( x ) at k iterate and β k is a scalar describing the attributes of the CG methods.

Some well-known formulas for the scalar β k are the Hestenes-Stiefel (HS) [5], Fletcher-Reeves (FR) [6], Polak-Ribière and Polak (PRP) [7], and Dai-Yuan (DY) [8] given by

β k H S = ∇ h k T y k d k − 1 T y k , β k P R P = ∇ h k T y k ‖ ∇ h k − 1 ‖ 2 ,

β k F R = ‖ ∇ h k ‖ 2 ‖ ∇ h k − 1 ‖ 2 , β k D Y = ‖ ∇ h k ‖ 2 d k − 1 T y k ,

where y k = ∇ h ( x k ) − ∇ h ( x k − 1 ) and ‖   ⋅   ‖ denotes the Euclidean norm. Due to their simplicity and low memory requirement, CG methods are more effective and desirable for large scale unconstrained smooth problems [9] [10]. The global convergence properties of nonlinear CG methods have been analyzed under the weak Wolfe line search

( h ( x k + α k d k ) ≤ h ( x k ) + ς α k ∇ h k T d k , ∇ h ( x k + α k d k ) T d k ≥ ρ ∇ h k T d k , (4)

and the strong Wolfe line search:

( h ( x k + α k d k ) ≤ h ( x k ) + ς α k ∇ h k T d k , | ∇ h ( x k + α k d k ) T d k | ≤ ρ | ∇ h k T d k | , (5)

where 0 < ς < ρ < 1 . CG methods use relatively little memory for large scale problems and require no numerical linear algebra, so each step is quite fast. However, they do not have second order information of the objective function, and typically converge much more slowly than Newton or quasi-Newton methods.

The quasi-Newton method is an iterative method with second order information of the objective function, and BFGS is the effective quasi-Newton method with the search direction

d k = − B k ∇ h k , (6)

where B k is an approximation of the Hessian matrix of h at x k . The update formula for B k is defined by

B k + 1 = B k − B k s k s k T s k T B k s k + y k y k T y k T s k , (7)

where s k is defined as s k = x k + 1 − x k , and the Hessian approximation B k + 1 of (7) satisfies the standard secant equation

B k + 1 s k = y k , (8)

if y k T s k > 0 , which is known as the curvature condition. The BFGS method has very interesting properties and remains one of the most respectable quasi-Newton methods for unconstrained optimization [11]. The theory of BFGS method and its global convergence have been established by many researchers (see [12]). For convex objective function, using some special inexact line search, it has been proved that the BFGS method is globally convergent (see [13] [14] [15]). However, when the objective function is nonconvex, the BFGS method under exact line search may fail to converge [16]. Moreover, Dai [17] proved that the BFGS method may fail for nonconvex functions with Wolfe line search techniques given in (4) and (5) [18]. Wolfe line search technique is the most common monotone line search technique, and it may leads to small steps without making significant progress to the minimum when the contours of the objective functions are a family of curves with large curvature (see [19] [20] [21] [22]). In order to overcome this drawback, the first nonmonotone line search technique was proposed by Grippo et al. [19] for Newton’s method. With this initiative, many nonmonotone line search techniques have been proposed in recent years [23]. Yuan et al. [24] developed a modified limited memory BFGS method with the update formula that has a higher order approximation to exact Hessian, and its convergence property is analyzed under the nonmonotone line search type. However, the method converges for only uniformly convex functions. Li et al. [25] proposed a new BFGS algorithm with modified secant equation which achieves both global and superlinear convergence for generally convex functions under the nonmonotone line search of [19]. Su and Rong [26] introduced and established a new spectral CG method and its implementation under a modified nonmonotone line search technique. They introduced a new spectral conjugate gradient direction

d k = ( − θ k ∇ h k + β k d k − 1     if   k ≥ 1 , − ∇ h 0               if   k = 0 , (9)

where

θ k = 1 + β k d k T ∇ h k ‖ ∇ h k ‖ 2 , (10)

β k = ∇ h k T y k − 1 ( 1 − τ ) ‖ ∇ h k − 1 ‖ 2 + τ d k − 1 T y k − 1 , (11)

and τ ∈ [ 0,1 ] . It is not difficult to notice that the denominator of (11) is the convex combination of the denominator of the conjugate parameters in HS and PRP conjugate gradient methods. The choice of spectral parameter given (10) ensures the sufficient descent property of the search direction without dependence of line search. The convergence property of their method analyzed under a new modified nonmonotone line search with some mild conditions. However, this spectral CG method has only first order information, and excludes second order information. When the number of dimension is large, the CG methods are more effective compared to the BFGS methods in term of the CPU-time but in term of the number of iterations and the number of function evaluations, the BFGS methods are better. In order to incorporate the remarkable properties of the CG and BFGS methods and to overcome their drawbacks, many hybrid of CG and BFGS methods are introduced for unconstrained smooth optimization [27] [28] [29]. However, the usage of these methods is mainly restricted to solve smooth optimization problems. Recently, Yuan et al. [30] [31] [32] [33] introduced some CG approaches to solve nonsmooth convex large scale problems using the smoothing regularization, and under some assumptions, the global convergence properties of these approaches are analyzed. Yuan and Wei [34] proposed the Barzilai and Borwein (BB) gradient method with nonmonotone line search to solve nonsmooth convex optimization problems. Some implementable quasi-Newton methods are also introduced for solving the same problem (see [35] [36] [37] [38]). More recently, Ou and Zhou [39] introduced a modified scaled BFGS preconditioned CG algorithm, and under appropriate assumptions, the method is proven to possess global convergence for nonsmooth convex functions.

Motivated by the work of Ou and Zhou [39], in this paper, we propose a hybrid approach of the a scaled CG method and a modified BFGS method to combine the simplicity of CG method and the Hessian approximation of BFGS method. Our work is mainly focused in developing the scaled conjugate search direction that includes the second order information of the objective function by incorporating the modified secant equation of BFGS method. Opposing the work of Ou and Zhou [39], our method has both the function and gradient value information of the objective function. Moreover, our method leads to better descent direction than the CG methods proposed so far. To the best of our knowledge, this is the first work to incorporate the scaled CG algorithm with the BFGS secant equation which contains both the function and gradient value information of the objective function for solving large scale nonsmooth optimization. Under the new modified nonmonotone line search technique, the global convergence of the algorithm is analyzed for nonsmooth convex problems.

The paper is organized as follows. In the next section, we consider a nonsmooth convex problem and review their basic results. In Section 3, we propose a new scaled CG algorithm that incorporates the BFGS secant equation which has both function value and gradient information of the objective function via the smoothing regularization. Using the new modified nonmonotone line search technique, we prove the global convergence of our new algorithm for nonsmooth convex problems. Numerical results and related comparisons are reported in Section 4. Finally, Section 5 concludes our work.

In this section, we consider the unconstrained optimization problem

min x ∈ R n f ( x ) , (12)

where f : R n → R is a possibly nonsmooth convex function. This problem is equivalent to the following problem

min x ∈ R n F ( x ) , (13)

where F : R n → R is the Moreau-Yosida regularization of f [40], which is defined by

F ( x ) = min z ∈ R n { f ( z ) + 1 2 λ ‖ z − x ‖ 2 } , (14)

where λ is a positive parameter. The function F is a finite-valued, continuously differentiable convex function even though the function f is nondifferentiable (see [40]). Let p ( x ) be the unique solution of (14). In what follows, we can express F ( x ) as

F ( x ) = f ( p ( x ) ) + 1 2 λ ‖ p ( x ) − x ‖ 2 . (15)

Moreover, the gradient of F is globally Lipschitz continuous, i.e.,

‖ g ( x ) − g ( y ) ‖ ≤ 1 λ ‖ x − y ‖ , ∀ x , y ∈ R n , (16)

where

g ( x ) = ∇ F ( x ) = x − p ( x ) λ . (17)

The point x ∈ R n is an optimal solution to (12) if and only if g ( x ) = 0 (see [40]). Furthermore, under reasonable conditions the gradient of F is semismooth and some of its remarkable properties are given in [41] [42].

Several methods have been proposed to solve (13) by incorporating bundle methods and quasi-Newton methods ideas [43] [44] [45], but it is burdensome to evaluate the exact value of p ( x ) at any given point x [46]. Luckily, for each x ∈ R n and any ε > 0 , we can have p α ( x , ε ) ∈ R n such that

f ( p α ( x , ε ) ) + 1 2 λ ‖ p α ( x , ε ) − x ‖ 2 ≤ F ( x ) + ε . (18)

Therefore, we can approximate F ( x ) and g ( x ) by

F α ( x , ε ) = f ( p α ( x , ε ) ) + 1 2 λ ‖ p α ( x , ε ) − x ‖ 2 , (19)

and

g α ( x , ε ) = x − p α ( x , ε ) λ , (20)

respectively. Implementable algorithms to define such a p α ( x , ε ) for nonsmooth convex model can be seen in [47]. The noticeable attributes of F α ( x , ε ) and g α ( x , ε ) by the following proposition [48].

Proposition 1. Let p α ( x , ε ) be a vector that satisfies (18), and let F α ( x , ε ) and g α ( x , ε ) be defined by (19) and (20), respectively. Then we obtain

F ( x ) ≤ F α ( x , ε ) ≤ F ( x ) + ε , (21)

‖ p α ( x , ε ) − p ( x ) ‖ ≤ 2 λ ε , (22)

and

‖ g α ( x , ε ) − g ( x ) ‖ ≤ 2 ε / λ . (23)

Proposition 1 shows that the approximations of F α ( x , ε ) and g α ( x , ε ) can be made arbitrarily close to the exact values of F ( x ) and g ( x ) respectively.

In this section, we introduce the new scaled CG search direction that incorporates the modified BFGS secant equation, and then describe the new algorithm for solving nonsmooth problems. We make use of a modified nonmonotone line search technique introduced by [23] to compute a step size. Based on the above approximations, we redefine the search direction of CG method (3) to solve problem (13) as follows:

d k + 1 = ( − g α ( x k + 1 , ε k + 1 ) + β k + 1 d k     if   k ≥ 1 , − g α ( x k + 1 , ε k + 1 )               if   k = 0 , (24)

where ε is an appropriately chosen positive number. Ou and Zhou [39] provided a search direction defined by

d k + 1 = ( − Q ˜ k + 1 g α ( x k + 1 , ε k + 1 )     if   k   ≥ 1 , − g α ( x k + 1 , ε k + 1 )           if   k   = 0 , (25)

where Q ˜ k + 1 ∈ R n × n is defined

Q ˜ k + 1 = θ ˜ k + 1 I − θ ˜ k + 1 w k s k T + s k w k T w k T s k + [ 1 + θ ˜ k + 1 w k T w k w k T s k ] s k s k T w k T s k , (26)

with

θ ˜ k + 1 = s k T s k w k T s k ,

where

w k = y k ∗ + t k s k . (27)

The vector y k ∗ and t k in (27) are defined as

y k ∗ = g α ( x k + 1 , ε k + 1 ) − g α ( x k , ε k ) (28)

and

t k = t + max { s k T y k ‖ s k ‖ 2 , 0 } ( t > 0 ) . (29)

It is easy to observe that the (27) has only gradient value information. In order to have both gradient and function value information, we replace (27) and (29) by

w k ∗ = y k ∗ + max { t k ∗ , 0 } s k , (30)

and

t k ∗ = 6 [ F ( x k ) − F ( x k + α k d k ) ] + 3 ( g α ( x k + α k d k , ε k + 1 ) + g α ( x k , ε k ) ) T s k ‖ s k ‖ 2 , (31)

respectively. Thus, the BFGS method with the secant equation

B k + 1 s k = w k ∗ , (32)

and the update formula

B k + 1 = B k − B k s k s k T s k T B k s k + w k ∗ w k ∗ T w k ∗ T s k , (33)

has both gradient and function value information, and the matrix B k + 1 inherits the positive definiteness of B k for generally convex functions. Using the secant Equation (32), we propose the new search direction is defined by

d k + 1 = ( − θ ¨ k + 1 g α ( x k + 1 , ε k + 1 ) + β k + 1 d k − ϑ k + 1 w k ∗ ,   if   k ≥ 1, − g α ( x k + 1 , ε k + 1 ) ,                         if   k = 0, (34)

where

θ ¨ k + 1 = 2 − d k T g α ( x k + 1 , ε k + 1 ) ‖ g α ( x k + 1 , ε k + 1 ) ‖ 2 ( g α ( x k + 1 , ε k + 1 ) T w k ∗ ‖ d k ‖ ‖ w k ∗ ‖ ) , (35)

β k + 1 = g α ( x k + 1 , ε k + 1 ) T w k ∗ ‖ d k ‖ ‖ w k ∗ ‖ + | d k T y k ∗ | , (36)

and

ϑ k + 1 = d k T g α ( x k + 1 , ε k + 1 ) ‖ d k ‖ ‖ w k ∗ ‖ . (37)

Now, based on the above search direction, we describe our new scaled CG algorithm with a modified nonmonotone line search for solving problem (13) as follows.

Algorithm 1

Step 0. Given ϵ > 0 , β ∈ ( 0 , 1 ) , ε ∈ ( 0 , 1 ) , σ ∈ ( 0 , 1 ) , and a point x 0 ∈ R n . Set d 0 = − g α ( x 0 , ε 0 ) and k : = 0 .

Step 1. If ‖ g α ( x k , ε k ) ‖ < ϵ , then stop, else go to the next step.

Step 2. Compute the search direction d k by using (34)-(37).

Step 3. Set trial step size α k = 1 .

Step 4. Set x k + 1 = x k + α k d k and choose a scalar ε k + 1 such that 0 < ε k + 1 < ε k .

Step 5. Let μ ∈ ( 0,1 ] , M ≥ 1 is a positive integer, define m ( k ) = min { k + 1 , M } , and choose

μ k i ≥ μ , i = 0 , 1 , 2 , ⋯ , m ( k ) − 1 , ∑ i = 0 m ( k ) − 1 μ k i = 1.

Let α k ≥ 0 be bounded above and satisfy:

F α ( x k + α k d k , ε k + 1 ) ≤ max [ F α ( x k , ε k ) , ∑ i = 0 m ( k ) − 1 μ k i F α ( x k − i , ε k − i ) ] σ α k g α ( x k , ε k ) T d k . (38)

If (38) does not holds, define α k = β α k and go to step 5.

Step 6. Set K := k + 1 and go to step 1.

It can be observed that the line search technique in step 5 of Algorithm 1 is a nonmonotone line search technique with some modifications.

In this subsection, we establish the global convergence of our method for nonsmooth convex problem (12). To prove the global convergence of Algorithm 1, the following Lemmas are needed.

Lemma 1. Assume that the search direction d k is generated by Algorithm 1, then for all k ≥ 0 , we have

g α ( x k + 1 , ε k + 1 ) T d k + 1 ≤ − ‖ g α ( x k + 1 , ε k + 1 ) ‖ 2 , (39)

and

‖ d k + 1 ‖ ≤ 5 ‖ g α ( x k + 1 , ε k + 1 ) ‖ . (40)

Proof. If k = 0 , then

g α ( x 0 , ε 0 ) T d 0 = − g α ( x 0 , ε 0 ) T g α ( x 0 , ε 0 ) = − ‖ g α ( x 0 , ε 0 ) ‖ 2 ,

and

‖ d 0 ‖ = ‖ − g α ( x 0 , ε 0 ) ‖ ≤ ‖ g α ( x 0 , ε 0 ) ‖ ≤ 5 ‖ g α ( x 0 , ε 0 ) ‖ .

Let k ≥ 1 , then from (34) we have

g α ( x k + 1 , ε k + 1 ) T d k + 1 = − θ ¨ k + 1 ‖ g α ( x k + 1 , ε k + 1 ) ‖ 2 + β k + 1 g α ( x k + 1 , ε k + 1 ) T d k − ϑ k + 1 g α ( x k + 1 , ε k + 1 ) T w k ∗ ≤ − θ ¨ k + 1 ‖ g α ( x k + 1 , ε k + 1 ) ‖ 2 + g α ( x k + 1 , ε k + 1 ) T w k ∗ ‖ d k ‖ ‖ w k ∗ ‖ + | d k T y k ∗ | g α ( x k + 1 , ε k + 1 ) T d k       − d k T g α ( x k + 1 , ε k + 1 ) ‖ d k ‖ ‖ w k ∗ ‖ g α ( x k + 1 , ε k + 1 ) T w k ∗ = − 2 ‖ g α ( x k + 1 , ε k + 1 ) ‖ 2 + d k T g α ( x k + 1 , ε k + 1 ) ‖ d k ‖ ‖ w k ∗ ‖ g α ( x k + 1 , ε k + 1 ) T w k ∗

      + g α ( x k + 1 , ε k + 1 ) T w k ∗ ‖ d k ‖ ‖ w k ∗ ‖ + | d k T y k ∗ | g α ( x k + 1 , ε k + 1 ) T d k       − d k T g α ( x k + 1 , ε k + 1 ) ‖ d k ‖ ‖ w k ∗ ‖ g α ( x k + 1 , ε k + 1 ) T w k ∗ = − 2 ‖ g α ( x k + 1 , ε k + 1 ) ‖ 2 + g α ( x k + 1 , ε k + 1 ) T w k ∗ ‖ d k ‖ ‖ w k ∗ ‖ + | d k T y k ∗ | g α ( x k + 1 , ε k + 1 ) T d k ≤ − 2 ‖ g α ( x k + 1 , ε k + 1 ) ‖ 2 + ( ‖ g α ( x k + 1 , ε k + 1 ) ‖ 2 ‖ d k ‖ ‖ w k ∗ ‖ ) ‖ d k ‖ ‖ w k ∗ ‖ = − ‖ g α ( x k + 1 , ε k + 1 ) ‖ 2 .

Once more, (34) yields that

‖ d k + 1 ‖ = ‖ − θ ¨ k + 1 g α ( x k + 1 , ε k + 1 ) + β k + 1 d k − ϑ k + 1 w k ∗ ‖ = ‖ − θ ¨ k + 1 g α ( x k + 1 , ε k + 1 ) + g α ( x k + 1 , ε k + 1 ) T w k ∗ ‖ d k ‖ ‖ w k ∗ ‖ + | d k T y k ∗ | d k − d k T g α ( x k + 1 , ε k + 1 ) ‖ d k ‖ ‖ w k ∗ ‖ w k ∗ ‖ ≤ ‖ θ ¨ k + 1 g α ( x k + 1 , ε k + 1 ) ‖ + 2 ‖ g α ( x k + 1 , ε k + 1 ) ‖ ≤ 4 ‖ g α ( x k + 1 , ε k + 1 ) ‖ + ( ‖ d k ‖ ‖ w k ∗ ‖ ‖ g α ( x k + 1 , ε k + 1 ) ‖ 2 ) ‖ g α ( x k + 1 , ε k + 1 ) ‖ 3 ‖ d k ‖ ‖ w k ∗ ‖ ≤ 5 ‖ g α ( x k + 1 , ε k + 1 ) ‖ .

Thus, the proof is completed.

Lemma 1 shows that the search direction d k developed in (34)-(37) leads to the most sufficiently descent direction and it belongs to a trust region.

Lemma 2. Let the step size α k satisfy (38), then there exist β > 0 satisfy a

α k ≥ min { 1, ( 1 − σ ) β L | g α ( x k , ε k ) T d k | ‖ d k ‖ 2 } . (41)

Proof. If α k = 1 satisfies the formula (38), then the proof is completed. Otherwise, there exist β such that

F α ( x k + α k β d k , ε k + 1 ) > max { F α ( x k , ε k ) , ∑ i = 0 m ( k ) − 1 μ k i F α ( x k − i , ε k − i ) } + σ α k β g α ( x k , ε k ) T d k > F α ( x k , ε k ) + σ α k β g α ( x k , ε k ) T d k .

Thus,

F α ( x k + α k β d k , ε k + 1 ) − F α ( x k , ε k ) > σ α k β g α ( x k , ε k ) T d k . (42)

Using mean value theorem, we have

F α ( x k + α d k , ε k + 1 ) − F α ( x k , ε k ) = ∫ 0 α ( g α ( x k + t d k , ε k + 1 ) − g α ( x k , ε k ) ) T d k d t + α g α ( x k , ε k ) T d k ≤ 1 2 L α 2 ‖ d k ‖ 2 + α g α ( x k , ε k ) T d k .

Combining the above inequality with (42), we have

α k ≥ min { 1, ( 1 − σ ) β L | g α ( x k , ε k ) T d k | ‖ d k ‖ 2 } .

Thus, the proof is completed.

Lemma 3. Assume that the sequence { x k } is generated by Algorithm 1, then we have

F α ( x k , ε k ) ≤ F α ( x 0 , ε 0 ) + μ σ ∑ i = 0 k − 2     α i g α ( x i , ε i ) T d i + σ α k − 1 g α ( x k − 1 , ε k − 1 ) T d k − 1 ≤ F α ( x 0 , ε 0 ) + μ σ ∑ i = 0 k − 1     α r g α ( x i , ε i ) T d i .

Proof. We prove this lemma by induction. For k = 1 , by (38) and μ ≤ 1 , we have

F α ( x 1 , ε 1 ) ≤ F α ( x 0 , ε 0 ) + σ α 0 g α ( x 0 , ε 0 ) T d 0 ≤ F α ( x 0 , ε 0 ) + μ σ α 0 g α ( x 0 , ε 0 ) d 0

Assume the equation holds for 1 , 2 , ⋯ , k , and we need to show for k + 1 . To show the condition, we have considered two cases.

Case 1:

max [ F α ( x k , ε k ) , ∑ i = 0 m ( k ) − 1 μ k i F α ( x k − i , ε k − i ) ] = F α ( x k , ε k ) .

Then, from (38), we have

F α ( x k + 1 , ε k + 1 ) = F α ( x k + α k d k , ε k + 1 ) ≤ F α ( x k , ε k ) + σ α k g α ( x k , ε k ) T d k ≤ F α ( x 0 , ε 0 ) + μ σ ∑ i = 0 k − 1     α i g α ( x i , ε i ) T d i + σ α k g α ( x k , ε k ) T d k ≤ F α ( x 0 , ε 0 ) + μ σ ∑ i = 0 k     α i g α ( x i , ε i ) T d i .

Case 2:

max [ F α ( x k , ε k ) , ∑ i = 0 m ( k ) − 1 μ k i F α ( x k − i , ε k − i ) ] = ∑ i = 0 m ( k ) − 1 μ k i F α ( x k − i , ε k − i ) ,

let n = min [ k , m − 1 ] . Then, again from (38),

F α ( x k + 1 , ε k + 1 ) = F α ( x k + α k d k , ε k + 1 ) ≤ ∑ j = 0 n     μ k j F α ( x k − j , ε k − j ) + σ α k g α ( x k , ε k ) T d k ≤ ∑ j = 0 n     μ k j [ F α ( x 0 , ε 0 ) + μ σ ∑ i = 0 k − j − 2 α i g α ( x i , ε i ) T d i     + σ α k − j − 1 g α ( x k − j − 1 , ε k − j − 1 ) T d k − j − 1 ] + σ α k g α ( x k , ε k ) T d k .

Thus, by imposing

( 1,2, ⋯ , n ) × ( 1,2, ⋯ , k − n − 2 ) ⊂ { ( j , i ) : 0 ≤ j ≤ n ,0 ≤ i ≤ k − n − 2 } ,

and

∑ j = 0 n     μ k j = 1 , μ k j ≥ μ ,

we have

F α ( x k + 1 , ε k + 1 ) ≤ F α ( x 0 , ε 0 ) + μ ∑ i = 0 k − n − 2 ( ∑ j = 0 n     μ k j ) α i g α ( x i , ε i ) T d i       + σ ∑ j = 0 n     μ k j α k − j − 1 g α ( x k − j − 1 , ε k − j − 1 ) T d k − j − 1 + σ α k g α ( x k , ε k ) T d k

≤ F α ( x 0 , ε 0 ) + μ σ ∑ i = 0 k − n − 2 α i g α ( x i , ε i ) T d i       + μ σ ∑ i = k − j − 1 k − 1 α i g α ( x i , ε i ) T d i + σ α k g α ( x k , ε k ) T d k = F α ( x 0 , ε 0 ) + μ σ ∑ i = 0 k − 1     α i g α ( x i , ε i ) T d i + σ α k g α ( x k , ε k ) T d k ≤ F α ( x 0 , ε 0 ) + μ σ ∑ i = 0 k     α i g α ( x i , ε i ) T d i .

Theorem 1. Assume that the sequences { x k } and { d k } are generated by Algorithm 1. Let F is bounded below on the level set L 0 = { x ∈ R n | F ( x ) ≤ F ( x 0 ) } and

lim k → ∞ ε k = 0.

Then

lim k → ∞ g α ( x k , ε k ) T d k = 0. (43)

Proof. Suppose that (43) is not true. Then there exist constants γ > 0 and k 0 such that

g α ( x k , ε k ) T d k ≤ − γ , ∀ k > k 0 . (44)

From Lemma 3, we have

F α ( x 0 , ε 0 ) − F α ( x k , ε k ) ≥ − μ σ ∑ i = 0 k − 1     α r g α ( x i , ε i ) T d i . (45)

By (40), (41) and (44), we have

F α ( x 0 , ε 0 ) − F α ( x k , ε k ) ≥ − μ σ ∑ i = 0 k − 1     α i g α ( x i , ε i ) T d i ≥ μ σ γ ∑ i = 0 k − 1     α i ≥ μ σ γ ∑ i = 0 k − 1 min { 1 , ( 1 − σ ) β L | g α ( x k , ε k ) T d k | ‖ d k ‖ 2 } ≥ μ σ γ ∑ i = 0 k − 1 min { 1 , ( 1 − σ ) β 25 L } .

Letting k → ∞ , we have

μ σ γ ∑ k = 0 ∞ min { 1 , ( 1 − σ ) β 25 L } ≤ ∑ k = 0 ∞     F α ( x 0 , ε 0 ) − F α ( x k , ε k ) ,

and this contradicts our assumption on F. Hence the theorem is proved.

Theorem 2. Let the conditions in Lemma 1 and Theorem 1 hold, then Algorithm 1 converges for nonsmooth problem (12).

Proof. From Lemma 1 and Theorem 1, we have

0 ≥ lim k → ∞ ( − ‖ g α ( x k , ε k ) ‖ 2 ) ≥ lim k → ∞ g α ( x k , ε k ) T d k = 0.

Then,

lim k → ∞ ‖ g α ( x k , ε k ) ‖ = 0. (46)

Thus, (23) and convergence of sequence { ε k } yield

0 ≤ lim k → ∞ ‖ g α ( x k , ε k ) − g ( x k ) ‖ ≤ lim k → ∞ 2 ε / λ = 0.

Hence,

lim k → ∞ ‖ g ( x k ) ‖ = 0. (47)

Let x ∗ be an accumulation point of { x k } . Then there exists a subsequence { x k } K satisfying

lim k ∈ K , k → ∞ x k = x ∗ . (48)

Thus, (17), (43) and (47) yield x ∗ = p ( x ∗ ) . Therefore x ∗ is an optimal solution of nonsmooth problem (12).

In this section, we present some numerical experiments to examine the efficiency of Algorithm 1 for some large scale nonsmooth academic test problems which are introduced in [49]. The details of these large scale nonsmooth academic test problems with their initial points x i ( 1 ) and the minimum values f ( x ∗ ) are listed as follows:

Problem 1

f ( x ) = max 1 ≤ i ≤ n x i 2

x i ( 1 ) = i for i = 1 , ⋯ , n / 2 and

x i ( 1 ) = − i for i = n / 2 + 1 , ⋯ , n

f ( x ∗ ) = 0.

Problem 2

f ( x ) = max 1 ≤ i ≤ n | ∑ i = 1 n x j i + j − 1 |

x i ( 1 ) = i for i = 1, ⋯ , n .

f ( x ∗ ) = 0.

Problem 3

f ( x ) = ∑ i = 1 n − 1 max { − x i − x i + 1 , − x i − x i + 1 + ( x i 2 + x i + 1 2 − 1 ) }

x i ( 1 ) = − 0.5 for i = 1, ⋯ , n ;

f ( x ∗ ) = − 2 ( n − 1 ) .

Problem 4

f ( x ) = ∑ i = 1 n − 1 max { x i 4 + x i + 1 2 , ( 2 − x i ) 2 + ( 2 − x i + 1 ) 2 , 2 e − x i + x i + 1 }

x i ( 1 ) = 2 for i = 1, ⋯ , n ;

f ( x ∗ ) = 2 ( n − 1 ) .

Problem 5

f ( x ) = max { ∑ i = 1 n − 1 ( x i 4 + x i + 1 2 ) , ∑ i = 1 n − 1 ( ( 2 − x i ) 2 + ( 2 − x i + 1 ) 2 ) , ∑ i = 1 n − 1 ( 2 e − x i + x i + 1 ) }

x i ( 1 ) = 2 for i = 1, ⋯ , n ;

f ( x ∗ ) = 2 ( n − 1 ) .

Problem 6

f ( x ) = max 1 ≤ i ≤ n { g ( − ∑ i = 1 n     x i ) , g ( x i ) } ,

where g ( y ) = ln ( | y | + 1 ) ;

x i ( 1 ) = 1 for i = 1, ⋯ , n ;

f ( x ∗ ) = 0.

Problem 7

f ( x ) = ∑ i = 1 n − 1 ( | x i | x i + 1 2 + 1 + | x i + 1 | x i 2 + 1 )

x i ( 1 ) = − 1 when mod ( i , 2 ) = 1 , ( i = 1 , ⋯ , n ) and

x i ( 1 ) = 1 when mod ( i , 2 ) = 0 , ( i = 1 , ⋯ , n ) ;

f ( x ∗ ) = 0.

Problem 8

f ( x ) = ∑ i = 1 n − 1 ( − x i + 2 ( x i 2 + x i + 1 2 − 1 ) + 1.75 | x i 2 + x i + 1 2 − 1 | )

x i ( 1 ) = − 1 for i = 1, ⋯ , n ;

f ( x ∗ ) = v a r i e s .

Problem 9

f ( x ) = max { ∑ i = 1 n − 1 ( x i 2 + ( x i + 1 − 1 ) 2 + x i + 1 − 1 ) , ∑ i = 1 n − 1 ( − x i 2 − ( x i + 1 − 1 ) 2 + x i + 1 + 1 ) }

x i ( 1 ) = − 1.5 when mod ( i , 2 ) = 1 , ( i = 1 , ⋯ , n ) and

x i ( 1 ) = 2.0 when mod ( i , 2 ) = 0 , ( i = 1 , ⋯ , n ) ;

f ( x ∗ ) = 0.

Problem 10

f ( x ) = ∑ i = 1 n − 1 max { x i 2 + ( x i + 1 − 1 ) 2 + x i + 1 − 1 , − x i 2 − ( x i + 1 − 1 ) 2 + x i + 1 + 1 }

x i ( 1 ) = − 1.5 when mod ( i , 2 ) = 1 , ( i = 1 , ⋯ , n ) and

x i ( 1 ) = 2.0 when mod ( i , 2 ) = 0 , ( i = 1 , ⋯ , n ) ;

f ( x ∗ ) = 0.

The problems 1 - 5 are convex functions, and the others are nonconvex functions. We test the above problems with the dimension of n = 1000 , n = 3000 , n = 5000 , n = 6000 , n = 10000 , n = 12000 , n = 20000 , n = 50000 , n = 60000 and n = 100000 . For convenience sake, we denote Algorithm 1 by scaled conjugate gradient method based on modified secant equation of BFGS method (SCG-MBFGS), and in order to demonstrate validity of our algorithm, we also list the results of other three algorithms MPRP in [30], MHS in [31] and MSBFGS-CG in [39]. All algorithms were implemented in Fortran 90 and run on a PC with an intel(R) Core(TM)i3-3110M CPU at 2.40 GHz, 4.00 GB of RAM, and the Windows 7 operating system. We stopped the iteration when the condition ‖ g α ( x k , ε k ) ‖ ≤ 10 − 10 was satisfied. The parameters for SCG-MBFGS were chosen as M = 10 β = 0.6 , σ = 0.85 λ = μ = 1 . All parameters for other three methods are chosen as in [30] [31] and [39] respectively. Table 1 shows the numerical results of SCG-MBFGS, MPRP, MHS and MSBFGS-CG on the given test problems. The columns in Table 1 have the following meanings:

Dim: the dimensions of problem.

NI: the total number of iterations.

NF: the number of function evaluations.

TIME: the CPU time in seconds.

f ( x ) : the value of f ( x ) at the final iteration.

From the numerical results in Table 1, it is not difficult to see that

SCG-MBFGS is superior or competitive to the other three methods in solving the given problems in terms of number of iteration, number of function evaluations and CPU time. Furthermore, to directly illustrate the performances of our method, we employed the tool provided by Dolan and Moré [50] to analyze and compare the efficiency of the method in terms of the number of iterations, number of function evaluations and CPU time. Figures 1-3 represent the computational performance profiles of the above algorithms regarding the number of iterations, number of function evaluations and CPU time respectively. From the 3 figures, we can observe that for the given test problems, SCG-MBFGS is competitive or superior to other three methods in terms of number of iteration, function evaluations and CPU time respectively.

From Figure 1 and Figure 2, we also notice that SCG-MBFGS performs better than the other methods do in terms of the numbers of iterations and function evaluations. Figure 3 indicates that MHS is comparable to SCG-MBFGS in terms of CPU time, and since the search direction of MHS is developed with only first order information while SCG-MBFGS, MPRP and MSBFGS-CG are with second order information, it is reasonable to need less CPU time for MHS.

In this paper, we propose a new scaled conjugate gradient method which incorporates a modified secant equation of BFGS method. This modified secant equation contains both function value and gradient information of the objective function, and its Hessian approximation update generates positive definite matrix. Under a modified nonmonotone line search and some mild conditions, the strong global convergence of the proposed method is analyzed for nonsmooth convex problems. The search direction of our new method generates sufficiently descent condition and belongs to a trust region. Compared with existing nonsmooth CG methods, the search direction of our approach is more descent direction. Numerical results and related comparisons show that the proposed method is effective for solving large scale nonsmooth optimization problems.

The authors would like to thank the reviewers and editor for their valuable comments which greatly improve our paper. This work is supported by the National Natural Science Foundation of China [Grant No. 11771003].

The authors declare no conflicts of interest regarding the publication of this paper.

Woldu, T.G., Zhang, H.B. and Fissuh, Y.H. (2020) A Scaled Conjugate Gradient Method Based on New BFGS Secant Equation with Modified Nonmonotone Line Search. American Journal of Computational Mathematics, 10, 1-22. https://doi.org/10.4236/ajcm.2020.101001