SVM (Support Vector Machine) [1] is a supervised learning algorithm commonly used for classification tasks and has been successfully applied to many technological fields, such as text categorization [2] , financial forecast [3] , image classification [4] and so on. This paper focuses on a binary classification problem. Given training samples { ( x i , y i ) , i = 1 , ⋯ , m } , where x i ∈ ℝ n , y i ∈ { − 1,1 } , the objective of SVM is to identify an optimal separating hyperplane to separate data points into two classes. Scholars have proposed some classic SVM models based on convex loss functions, such as the hinge loss (also called L₁ loss) in classic SVM [5] , the least square loss in LSSVM [6] and the huberized pinball loss in HPSVM [7] . However, in practice, the real dataset often contain noise. Since convex loss functions are generally unbounded, convex losses are highly sensitive to outliers and potentially influenced by outliers. Therefore, some nonconvex loss functions are proposed to improve robustness compared with the convex loss functions [8] . For example, [9] proposed the ramp loss based on hinge loss, the truncated pinball loss was proposed by [10] . Recently, a noise insensitive and robust support vector machine classifier with huberied truncated pinball loss (HTPSVM) was proposed in [11] , this loss is smooth and nonconvex loss function. The HTPSVM can be transformed into format of “Loss + Penalty”, in which the penalty is a hybrid of l 1 norm and l 2 norm penalty.

Here, the HTPSVM model and algorithm of literature [11] are briefly introduced. Consider a classification problem with training samples { x i , y i } i = 1 m ⊂ ℝ d × { − 1,1 } . The HTPSVM seeks to solve the following regularization problem:

min b ∈ ℝ , w ∈ ℝ d 1 m ∑ i = 1 m   l h t p ( y i ( b + w T x i ) ) + λ ‖ w ‖ 1 + ‖ w ‖ 2 2 2 + b 2 2 , (1)

the huberied truncated pinball loss l h t p ( ⋅ ) function is defines as

l h t p ( u ) = { 1 , u ≤ − 2 5 4 5 − u − 5 4 u 2 , − 2 5 < u ≤ 0 , 4 5 − u , 0 < u ≤ 3 5 , 5 4 ( 1 − u ) 2 , 3 5 < u ≤ 1 , 5 8 ( 1 − u ) 2 , 1 ≤ u < 7 5 , − 1 2 ( 6 5 − u ) , 7 5 ≤ u < 8 5 , − 1 2 ( 6 5 − u ) − 5 8 ( u − 8 5 ) 2 , 8 5 ≤ u < 2 , 3 10 , u ≥ 2 , (2)

which is a nonconvex and smooth function. The HTPSVM combine the benefits of both l 1 and l 2 norm regularizers and and it has been demonstrated in [11] that it can reduce the effects of noise in the training sample. Therefore, we consider that studying the HTPSVM model is meaningful. The APG algorithm was used to solve the model in [11] . [12] applied the APG_s method (first proposed in [13] ) to solve problem (1) and obtain better convergence behavior. However we find that the proximal operator for computing the l 1 norm causes the subproblem to be solved slowly in APG and APG_s algorithms, we attempt to accelerate the solution process for this model. Recently, [14] propose the Bregman APG_s (BAPG_s) method, which avoids the restrictive global Lipschitz gradient continuity assumption. In this paper, we improve BAPG_s algorithm to solve the problem (1) and replace the Lipschitz constant by an appropriate positive definite matrix and obtain better results after we perform numerical experiments on 10 datasets to test our method.

The rest of this paper is organized as follows. In the next section, we provide preliminary materials used in this work. In Section 3, we introduce the BAPG_s algorithm proposed by [14] and present our algorithm based on the BAPG_s method for solving the HTPSVM model (1). The convergence of our method is also discussed. Section 4 performs some experiments.

In this paper, we let ℝ denote the set of real numbers. We work in the Euclidean space ℝ n , and the standard Euclidean inner product and the induced norm on ℝ n are denoted by 〈 ⋅ , ⋅ 〉 and ‖   ⋅   ‖ . The domain of the function f : ℝ n → ( − ∞ , + ∞ ] is defined by dom   f = { x ∈ ℝ n : f ( x ) < + ∞ } . We say that f is proper if dom   f ≠ ∅ . A proper function f is said to be closed if it is lower semicontinuous at any x ∈ dom   f , i.e. f ( x ) ≤ lim inf z → x f ( z ) .

Definition 1. [ [15] , Definition 8.3] For a proper closed function f, the regular subdifferential of f : ℝ n → ℝ ∪ { + ∞ } at x ∈ dom   f is defined by

∂ ^ f ( x ) : = { x ^ ∈ ℝ n : lim inf z → x , z ≠ x f ( z ) − f ( x ) − 〈 x ^ , z − x 〉 ‖ z − x ‖ ≥ 0 } . (3)

The (general) subdifferential of f at x ∈ dom   f is defined

∂ f ( x ) : = { x ^ :   ∃ x k → f   x , x ^ k → x ^   with   x ^ k ∈ ∂ ^ f ( x k )   for   each   k } , (4)

where x k → f   x means both x k → x and f ( x k ) → f ( x ) . Note that if f is also convex, then the general subdifferential and regular subdifferential of f at x ∈ dom   f reduce to the classical subdifferential [ [15] , Proposition 8.12], that is ∂ f ( x ) = { x ^ : f ( y ) ≥ f ( x ) + 〈 x ^ , y − x 〉   for   all   y } . (5)

Definition 2. (Kernel Generating Distances and Bregman Distances [16] [17] [18] ) Let C be a nonempty, convex and open subset of ℝ n . Associated with C, a function ϕ : ℝ n → ( − ∞ , + ∞ ] is called a kernel generating distance if it satisfies the following:

1) ϕ is proper, lower semicontinuous and convex, with dom   ϕ ⊂ C ¯ and dom   ∂ ϕ = C ;

2) ϕ is continuously differentiable on int dom   ϕ ≡ C .

We denote the class of kernel generating distances by G ( C ) . Given ϕ ∈ G ( C ) , the Bregman distance D ϕ : dom   ϕ × int dom   ϕ → ( 0 , + ∞ ] is defined by

D ϕ ( x , y ) : = ϕ ( x ) − ϕ ( y ) − 〈 ∇ ϕ ( y ) , x − y 〉 .

For exmple, when ϕ ( x ) = ‖ x ‖ 2 , then D ϕ ( x , y ) = ‖ x − y ‖ 2 . If ϕ ( x ) = x T A x , then D ϕ ( x , y ) = ( x − y ) T A ( x − y ) . In this article, the gradient Lipschitz continuity condition of the function f is no longer required, instead it is replaced by the L-smooth adaptive function of pair ( f , ϕ ) . The definition of L-smooth adaptable as follows.

Definition 3. A pair of functions ( f , ϕ ) , ϕ ∈ G ( C ) , f : ℝ n → ( − ∞ , + ∞ ] is a proper and lower semicontinuous function with dom   ϕ ⊂ dom   f and f is continuously differentiable on C = int dom   ϕ , is called L-smooth adaptable (L-smad) on C if there exists L > 0 such that L ϕ − f and L ϕ + f are convex on C.

Lemma 1. (Full Extended Descent Lemma [19] ) A pair of functions ( f , ϕ ) is L-smad on C = int dom   ϕ if and only if: | f ( x ) − f ( y ) − 〈 ∇ f ( y ) , x − y 〉 | ≤ L D ϕ ( x , y ) , ∀ x , y ∈ int dom   ϕ .

Definition 4. f : ℝ n → ℝ ∪ { + ∞ } is called μ-relative weakly convex to ϕ on C if there exists μ > 0 such that f + μ ϕ is convex on C [14] .

In this section, we first describe the BAPG_s method proposed in [14] , then the modified BAPG_s method with adaptive parameter is given for HTPSVM.

In this section, we aim to show the performance of Algorithm 2 for solving problem (1) by using MATLAB R2020b on a 64-bit PC with an Intel(R) Core(TM) i7-10870H CPU (2.20GHz) and 16GB of RAM.

First, consider the optimality condition (19) of Algorithm 2

0 ∈ ∇ f ( y k ′ ) − ξ k + ∂ P 1 ( z k + 1 ) + θ Q ( z k + 1 − z k ) = ∇ f ( y k ) − ξ k + λ ∂ ‖ z k + 1 ‖ 1 + z k + 1 + θ Q ( z k + 1 − z k ) .

Due to there is no explicit solution for this subproblem, we try to instead the l 1 norm by linear approximation, that is, ‖ z ‖ 1 ≈ ‖ x k ‖ 1 + v k T ( z − x k ) , where v k ∈ ∂ ‖ x k ‖ 1 (here we take v k : = sign ( x k ) ), then we construct a new iteration step to replace the subproblem in Algorithm 2 as

z k + 1 = arg min z ∈ ℝ n { 〈 ∇ f ( y k ) − ξ k , z − y k 〉 + ‖ x k ‖ 1 + λ v k T ( z − x k )     + 1 2 z T z + θ 2 [ ( z − z k ) T Q ( z − z k ) ] } , (20)

it is easy to calculate its solution:

0 = ∇ f ( y k ) − ξ k + λ v k T + z k + 1 + θ Q ( z k + 1 − z k ) ,

which means

( I + θ Q ) z k + 1 = θ Q z k − ∇ f ( y k ) + ξ k − λ v k .

Then the update (18) and (19) are replaced by

{ z k ′ + 1 = ( I + θ k ′ Q ) − 1 ( θ k ′ Q z k ′ − ∇ f ( y k ′ ) + ξ k ′ − λ v k ′ ) , z k + 1 = ( I + θ Q ) − 1 ( θ Q z k − ∇ f ( y k ) + ξ k − λ v k ) , (21)

in experiments, where v k ′ = sign ( x k ′ ) and v k = sign ( x k ) . The experiments are conducted on several real world datasets. We select 10 datasets from UCI [20] , to compare the Algorithm 2 with APG (method in [11] ), APG_s [12] and GIST [21] , where in GIST, we set F = f + P 1 with f = f 1 − f 2 − f 3 . The corresponding parameters of these methods are set the same as in [12] . For each dataset, The 21 initial points are used commonly for all methods: one zero vector, and 5 vectors selected independently from N (0, σ²I) for each σ ∈ { 1,2,4,8 } . All algorithms stop if ‖ ( b k + 1 ; w k + 1 ) − ( b k ; w k ) ‖ max { 1, ‖ ( b k ; w k ) ‖ } < 10 − 6 or the number of iterations hits 3000. The average results are given in Table 1 and Table 2, including the number of iterations (iter), objective function value (fval) and CPU time in seconds (CPU) at termination with λ = 1 × 10 − 3 and 5 × 10 − 4 , where BAPG_s- P 1 and APG_s- P 1 represent using BAPG_s (algorithm 2) and APG_s [12] for P 1 respectively ( P 1 described in section 3.2).

From the above tables, we see that Algorithm 2 for P 2 always obtain the smaller function values and converge faster than others, this means that Algorithm 2 for solving HTPSVM model (1) performs well.

In this paper, based on the BAPG_s method proposed by [14] , we construct the modified BAPG_s with the adaptive parameter selection technique introduced in [12] for solving the HTPSVM model. The linear approximation method is used to improve the subproblem in algorithm and a function ϕ with a suitable matrix Q is set to obtain the L-smad property. Finally, numerical experiments show that our algorithm convergence faster.

The author declares no conflicts of interest regarding the publication of this paper.

Ren, K.X. (2024) The Modified BAPG_s Method for Support Vector Machine Classifier with Truncated Loss. Applied Mathematics, 15, 267-278. https://doi.org/10.4236/am.2024.154015