Support vector machine (SVM) is a supervised learning method mainly used for classification problems [1] . It distinguishes data points of two different classes by finding an optimal hyperplane that maximizes the boundary between the two classes. SVM plays an important role in the field of machine learning for its excellent performance and generalization ability. Its applications span multiple subjects, including key areas such as text classification, image recognition, bioinformatics, and financial risk assessment.

The performance of SVM largely depends on the choice of its loss function. Different loss functions are suitable for different data distribution and noise conditions. The SVM with classic hinge loss (HSVM) [2] increases the distance between the nearest points of different categories, but the model is easily affected by noise. To overcome this limitation, the SVM with pinball losses (PSVM) [3] was proposed, which deducts points from correctly classified points and is insensitive to noise during resampling. However, it sacrifices sparsity and lacks flexibility in adjusting parameters that affect the loss value. In order to overcome these shortcomings, the SVM with generalized pinball loss (GPSVM) [4] was introduced. GPSVM reduces noise sensitivity while retaining sparsity and flexibility of parameter adjustment in the derived solution.

The above mentioned loss functions are convex but not differentiable, so the corresponding unconstrained minimization problems are not differentiable. Therefore, although some first-order and second-order optimization algorithms [5] are theoretically effective and have solid theoretical support, they cannot be directly used to solve such problems. In order to solve the limitation of traditional hinge loss function, researchers introduced smooth hinge loss function (SHSVM) [6] to improve the performance of the model. Similarly, to address the shortcomings of the pinball loss function, researchers proposed a smooth pinball loss function (SPSVM) [7] . Based on the above ideas, the smooth generalized pinball loss function (SGPSVM) [8] was given. These smooth loss functions are differentiable, so that the corresponding models can be solved directly by applying first order and second-order optimization techniques.

Although some progress has been made in the design of loss functions, the above mentioned loss functions are all unbounded, which makes the models face significant challenges when dealing with outlier data points. Outliers are those extreme values that differ significantly from the rest of the data set, and they can be caused by sampling errors or labeling errors. In classification tasks, it is critical to properly handle these outliers, as they can seriously affect the performance of the model. Many SVM models are particularly sensitive to these outliers because their loss functions may grow indefinitely on these points. This phenomenon has led to the development of bounded loss functions, such as the truncated hinge loss function in RSVM [9] . By truncating the loss function, the high values are reset to some constant(s), so that the impact of outliers is under control and the model performance is enhanced.

Recently, Zhu et al. [10] has proposed bounded huberied truncated pinball loss function which can reduce the effect of noise in training samples. Since quartic functions are more complex than quadratic functions, they may improve classifier performance, especially when data follows a nonlinear distribution. Based on this idea, we change the Huberized truncated pinball loss function by replacing the quadratic term with the quartic term. The newly proposed quartic truncated pinball function is bounded and differentiable. The Bregman modified second APG method by Wang et al. [11] performs well on such problems, so we use BAPGs to solve the support vector machine problem with the quartic truncated pinball function (QTPSVM).

The structure of this paper is as follows: In Section 2, some classical SVM models are reviewed. In Section 3, we introduce the QTPSVM model. Then we discuss the application of the BAPGs algorithm on QTPSVM in Section 4. Numerical experiments on real data are done to verify the effectiveness of QTPSVM in Section 5. The last Section is the summary of this paper.

In the SVM model for binary classification problem, we denote the training dataset as $\left\{\left(x_i,y_i\right)\in {ℝ}^n\times \left\{1,-1\right\},i=1,2,3,\cdots ,m\right\}$, where $x_i$ is a n-dimensional predictor variable, $y_i$ is its label and m is the number of samples. The key to solving this problem is to identify a separating hyperplane represented by $w^{\text{T}}x+b=0$, $w\in {ℝ}^n$, $b\in ℝ$, with the objective of maximizing the margin and the training error. The SVM classification can be expressed as an unconstrained optimization problem:

$\underset{w,b}{\text{min}}\frac{1}{m}\underset{i=1}{\overset{m}{{\displaystyle \sum }}}\ell \left(y_i\left(w^{\text{T}}{x}_i+b\right)\right)+{\text{Penalty}}_{\lambda }\left(w,b\right),$

where $\ell$ denotes the loss function and ${\text{Penalty}}_{\lambda }\left(w,b\right)$ is the penalty (regularization) term, $\lambda >0$ is the regularization parameter, used to balance the loss and the penalty. The decision function $w^{\text{T}}x+b$ is used to determine the class of $x$: $x$ is classified as positive if the the decision function is positive, and negative otherwise.

There are several kinds of penalty terms commonly used in support vector machine problems:

1) The $L_2$ norm penalty $\frac{\lambda }{2}{\Vert w\Vert }_2^2$. This is a traditional penalty term for various machine learning problems, but it does not perform well on sparse solutions.

2) The $L_1$ norm penalty $\lambda {\Vert w\Vert }_1$. It has a good effect on sparsity, but it is limited by the number of samples and may over-prune in some highly correlated features [12] .

3) The Elastic Net penalty combines the $L_1$ norm and the $L_2$ norm [13] , integrates the advantages of both terms and overcoming their drawbacks. The commonly used forms are:

${\lambda }_1{\Vert w_1\Vert }_1+\frac{{\lambda }_2}{2}{\Vert w_2\Vert }_2^2$ (2.1)

and

${\lambda }_1{\Vert w_1\Vert }_1+\frac{{\lambda }_2}{2}{\Vert w_2\Vert }_2^2+\frac{{\lambda }_3}{2}{b}^2.$ (2.2)

In SVM, the most popular and widely used loss function is the hinge loss function (Rectified Linear Unit function), which is denoted by ${\ell }_{\text{hinge}}:ℝ\to ℝ$, and is defined as follows:

${\ell }_{\text{hinge}}\left(u\right)=\text{max}\left\{0,1-u\right\}=\{\begin{array}{ll}1-u,\hfill & u\le 1,\hfill \\ 0,\hfill & u>1.\hfill \end{array}$

Another commonly used one is the pinball loss function ${\ell }_{\text{pinball}}:ℝ\to ℝ$,

${\ell }_{\text{pinball}}=\{\begin{array}{ll}1-u,\hfill & u\le 1,\hfill \\ \tau \left(u-1\right),\hfill & u>1.\hfill \end{array}$

where $\tau \ge 0$ is the preset parameter. It is obvious that ${\ell }_{\text{hinge}}$ is a special case of ${\ell }_{\text{pinball}}$ when $\tau =0$. The classification models obtained by combining the above two loss functions with the $L_2$-norm penalty are referred to as HSVM [2] and PSVM [3] respectively.

Both hinge loss function and pinball loss function are nondifferentiable (at zero point) convex functions. In applications, non-differentiability restricts the choice of algorithms, while convexity makes the loss function value of outliers larger, thus greatly affects the model effect.

In order to overcome this weakness of convexity, the truncation method is a common idea, which is to redefine large values as some constant (s). For example, the truncation hinge loss function in RSVM [9] is

$\ell \left(u\right)=\text{min}\left\{s,{\ell }_{\text{hinge}}\left(u\right)\right\}=\{\begin{array}{ll}s,\hfill & u\le 1-s,\hfill \\ 1-u,\hfill & 1-s<u\le 1,\hfill \\ 0,\hfill & u>1.\hfill \end{array}$

Such operation makes the model more stable when dealing with noisy data. However, the truncation method makes the loss function nonconvex, so it has higher requirements for the algorithms.

An important method for handling non-differentiability is to locally correct the function values near the non-differentiable points, to make the entire function differentiable. Huberization is a common correction approach that uses quadratic functions to handle non-differentiable points. For example, the Huberized hinge loss function used in HHSVM [14] :

${\ell }_H\left(u\right)=\{\begin{array}{ll}0,\hfill & u>1,\hfill \\ \frac{{\left(1-u\right)}^2}{2\delta },\hfill & 1-\delta <u\le 1,\hfill \\ 1-u-\frac{\delta }{2},\hfill & u\le 1-\delta ,\hfill \end{array}$

where $\delta >0$ is a pre-specified constant.

Based on the above methods, Zhu et al. [10] proposed the HTPSVM classification model:

$\underset{w,b}{\text{min}}\frac{1}{m}\underset{i=1}{\overset{m}{{\displaystyle \sum }}}{\ell }_{\text{htp}}\left(y_i\left(w^{\text{T}}{x}_i+b\right)\right)+{\lambda }_1{\Vert w\Vert }_1+\frac{{\lambda }_2}{2}{\Vert w\Vert }_2^2+\frac{{\lambda }_3}{2}{b}^2.$

where ${\lambda }_1,{\lambda }_2\ge 0$ are regularization parameters. The loss function is

${\ell }_{\text{htp}}\left(u\right)=\{\begin{array}{ll}{\theta }_1,\hfill & u\le 1-\delta -{\theta }_1,\hfill \\ 1-u-\frac{\delta }{2}-\frac{{\left(u-1+{\theta }_1\right)}^2}{2\delta },\hfill & 1-\delta -{\theta }_1<u\le 1-\delta ,\hfill \\ 1-u-\frac{\delta }{2},\hfill & 1-{\theta }_1<u\le 1-\delta ,\hfill \\ \frac{{\left(1-u\right)}^2}{2\delta },\hfill & 1-\delta <u\le 1,\hfill \\ \frac{\tau {\left(1-u\right)}^2}{2\delta },\hfill & 1<u\le 1+\delta ,\hfill \\ -\tau \left(1-u+\frac{\delta }{2}\right),\hfill & 1+\delta <u\le 1+{\theta }_2,\hfill \\ -\tau \left(1-u+\frac{\delta }{2}\right)-\frac{\tau {\left(u-1-{\theta }_2\right)}^2}{2\delta },\hfill & 1+{\theta }_2<u\le 1+\delta +{\theta }_2,\hfill \\ \tau {\theta }_2,\hfill & u>1+\delta +{\theta }_2.\hfill \end{array}$

where ${\theta }_1,{\theta }_2>0$, $\delta \in \left[0,1\min \left\{{\theta }_1,{\theta }_2\right\}\right)$, $\tau \in \left[0,1\right]$ are pre-specified constants. Here we note that ${\ell }_{\text{htp}}$ is obtained by truncating the pinball loss function and then applying Huberization. Experiments validate that HTPSVM not only has better performance in terms of noise resistance compared to other methods, but also demonstrates improved time efficiency in solving the problem using the APG algorithm.

Inspired by the Huberization method and HTPSVM, we replace the quadratic term in HTPSVM with a quartic function to obtain the following quartic truncated pinball loss function:

$l_{\text{qtp}}\left(u\right)=\{\begin{array}{ll}{\theta }_1{\delta }^2,\hfill & u\le 1-\delta -{\theta }_1,\hfill \\ -{\delta }^{2}u-\frac{3}{4}{\delta }^3+{\delta }^2-\frac{{\left(u-1+{\theta }_1\right)}^4}{4\delta },\hfill & 1-\delta -{\theta }_1<u\le 1-{\theta }_1,\hfill \\ -{\delta }^{2}u-\frac{3}{4}{\delta }^3+{\delta }^2,\hfill & 1-{\theta }_1<u\le 1-\delta ,\hfill \\ \frac{{\left(1-u\right)}^4}{4\delta },\hfill & 1-\delta <u\le 1,\hfill \\ \frac{\tau {\left(1-u\right)}^4}{4\delta },\hfill & 1<u\le 1+\delta ,\hfill \\ \tau {\delta }^{2}u-\frac{3}{4}\tau {\delta }^3-\tau {\delta }^2,\hfill & 1+\delta <u\le 1+{\theta }_2,\hfill \\ \tau {\delta }^{2}u-\frac{3}{4}\tau {\delta }^3-\tau {\delta }^2-\frac{\tau {\left(u-1-{\theta }_2\right)}^4}{4\delta },\hfill & 1+{\theta }_2<u\le 1+\delta +{\theta }_2,\hfill \\ \tau {\theta }_2{\delta }^2,\hfill & u>1+\delta +{\theta }_2,\hfill \end{array}$

where ${\theta }_1,{\theta }_2>0$, $0\le \delta \le \text{min}\left\{{\theta }_1,{\theta }_2\right\}$, $\tau \in \left[0,1\right]$ are pre-specified constants. We notice that ${\ell }_{\text{qtp}}\left(u\right)$ is bounded, continuous and differentiable, but not convex.

The SVM classifier with the quartic truncated pinball loss (QTPSVM) based on the elastic net penalty is defined as:

$\underset{w,b}{\text{min}}\frac{1}{m}\underset{i=1}{\overset{m}{{\displaystyle \sum }}}{\ell }_{\text{qtp}}\left(y_i\left(w^{\text{T}}{x}_i+b\right)\right)+{\lambda }_1{\Vert w\Vert }_1+\frac{{\lambda }_2}{2}{\Vert w\Vert }_2^2+\frac{{\lambda }_3}{2}{b}^2.$ (3.1)

where ${\lambda }_1,{\lambda }_2,{\lambda }_3$ are regularization parameters. The advantage of QTPSVM over HTPSVM lies in the fact that, compared to the uniform change of derivatives for quadratic functions, the derivative of a quartic function changes very slowly within each segment and more rapidly at the boundaries, with relatively smaller fluctuations in function values. This makes the impact of misclassified points near the separating hyperplane on the function value and derivative smaller, thereby enhances the noise resistance of the model. However, quartic functions may lead higher requirements on algorithms, so we need a suitable method to solve this problem, which is the BAPGS algorithm we introduce below.

The quartic truncated pinball loss is nonconvex. However, many algorithms for nonconvex optimization problems may not perform well on quartic functions due to the usage of the proximal operator.

So to solve (3.1), we adopt the Bregman modified second accelerated proximal gradient (BAPGs) method, which is a variant of the modified second accelerated proximal gradient (APGs) method by replacing the proximal operator by the minimization involving the Bregman distance.

To evaluate the performance of QTPSVM, we compare it with HTPSVM on real datasets. The HTPSVM is solved by the proximal gradient algorithm (APG) as in [20] . The experimental datasets are all taken from the UCI Machine Learning Repository [21] . Both models start from the zero vector for all experiments. The algorithm stops if both the following two conditions are met:

$\frac{F_{k-1}-F_k}{1+F_{k-1}}\le \text{tol},\frac{\Vert x^k-x^{k-1}\Vert }{1+\Vert x^{k-1}\Vert }\le \text{tol},$

where $x^k=\left(w^k,b^k\right)$, $F_k=F\left(x^k\right)$. The stopping tolerance $\text{tol}={10}^{-6}$.

As is well known, the choice of parameters plays a crucial role in the effectiveness of machine learning [22] - [24] . The parameters for the APG are set as in [10] . For QTPSVM and HTPSVM, we take ${\lambda }_1=5\times {10}^{-4}$, and $\tau =0.5$, and fix ${\lambda }_2={\lambda }_3=1$ because the algorithm seems not sensitive to ${\lambda }_2$ and ${\lambda }_3$ [10] . In HTPSVM, ${\theta }_1,{\theta }_2$, and $\delta$ are set according to the original paper, that is, they are tuned through 10-fold cross-validation from $\left\{0.1,0.3,0.5,0.7,0.9\right\}$. In QTPSVM, we found by tenfold cross-validation on the same parameter combination set that experiments perform best and most efficiently on most dataset. Other combinations are not only less effective in most cases but also time-consuming, so we fix this parameter combination. After determining the parameter combination, for each dataset, we randomly divide it into training and test sets 50 times; for each division, 80% of the dataset is used for training, and the remaining 20% is used for testing.

All numerical experiments were conducted on an Intel(R) Core(TM) i5-8265U processor with 8.00 GB memory. The operating system is Windows 10, and the version of Matlab used is R2021a.

We compare QTPSVM with HTPSVM. To further study the robustness of QTPSVM, we randomly flip the labels of the training set, with flip levels of 0%, 5%, and 10%. The classification accuracy on the test set (averaged over 50 repetitions) is recorded in Table 1 . We also record the average running time for each method (in seconds) in Table 2 . To further evaluate the performance of the classifiers, the area under the ROC curve (AUC) is calculated and recorded in Table 3 .

Through Tables 1-3 , we can observe that the average running time of QTPSVM is less than that of HTPSVM. In terms of classification accuracy, QTPSVM has higher accuracy and AUC on many datasets compared to HTPSVM. This improvement in performance remains stable even when different levels of label flipping are applied.

In this paper we propose a new robust support vector machine classification model QTPSVM, based on a quartic truncated pinball loss function, and solve it using the BAPGs method. By comparing the numerical results of QTPSVM and HTPSVM in real data, it is shown that this new classifier is effective and stable to noise. However, QTPSVM involves multiple parameters, and parameter selection requires cross-validation tuning, which increases the complexity and computational cost of model training. Therefore, QTPSVM may not be the most efficient choice for very large-scale datasets.

The authors declare they have not used Artificial Intelligence (AI) tools in the creation of this article.