Principal component analysis and generalized low rank approximation of matrices are two different dimensionality reduction methods. Two different dimensionality reduction algorithms are applied to the L1-CSVM model based on augmented Lagrange method to explore the variation of running time and accuracy of the model in dimensionality reduction space. The results show that the improved algorithm can greatly reduce the running time and improve the accuracy of the algorithm.
Support Vector Machine (SVM) [
Sparse representation optimization is widely used. L1 norm is usually used to regularize penalty terms to improve the classification accuracy of network models. The L1 norm uses fewer features for classification by selecting. L1 norm is considered to be an effective method to reduce the influence of outliers [
Based on the study of the above theory, this paper takes feature dimension reduction as the starting point to enhance the performance of image binary classification algorithm. PCA and GLRAM are introduced to improve the algorithm model of L1-CSVM based on ALM. The experimental results show that when calculating the L1-CSVM model based on dimensionality reduction, it can significantly reduce the calculation time and generally improve the accuracy due to the reduction of the number of data features.
The following contents of this article are as follows. In Section 2, PCA and GLRAM are introduced. In Section 3, PCA-L1-CSVM and GLRAM-L1-CSVM are described. In Section 4, the content and results of comparative experiments on MNIST and Fashion-MNIST datasets are introduced in detail. Finally, the conclusion of this paper is in Section 5.
This section mainly introduces two dimensionality reduction algorithms: PCA and GLRAM.
Given data T = { ( x 1 , x 2 , ⋯ , x m ) } ∈ R d × m , Let’s say we have m samples, and each sample corresponds to d attributes. Take any centralized data x ˜ i ∈ ℝ d , A new coordinate system composed of orthonormal basis vectors is obtained by projection transformation { v 1 , v 2 , ⋯ , v d ′ } ( d ′ ≤ d ) , ‖ v i ‖ 2 = 1 , v i T v j = 0 ( i ≠ j ) . The projection of x ˜ i in my new coordinates is s i = ( s i 1 ; s i 2 ; ⋯ ; s i d ′ ) , meet s i j = v j T x ˜ i and x ^ i = ∑ j = 1 d ′ s i j v j . Therefore, the following optimization model can be established
∑ i = 1 m ‖ ∑ j = 1 d ′ s i j v j − x ˜ i ‖ 2 2 s .t . ‖ v i ‖ 2 = 1 , v i T v j = 0 ( i ≠ j ) (1)
The orthogonal coordinate system can be obtained by solving (1), from which low-dimensional characteristic data s i = ( s i 1 ; s i 2 ; ⋯ ; s i d ′ ) ∈ R d ′ × m can be obtained. Thus, the original high-dimensional data set T is transformed into sparse representation data set T r = { ( r i , y i ) } i = 1 m ∈ R d ′ × { ± 1 } .
GLRAM minimizes the reconstruction error of the new matrix and the original matrix by reducing the dimension of the matrix data, and uses the low-rank matrix set to approximate the original matrix set to complete data compression. Given data T = { ( X i , y i ) } i = 1 m ∈ R r × c × { ± 1 } , Let’s say we have m samples, and each sample corresponds to r × c attributes. We want to find two orthogonal transformation matrices L ∈ R r × q 1 , P ∈ R c × q 2 and M i ∈ R q 1 × q 2 , i = 1 , ⋯ , m , make L M i P T as close as possible to X i .
The following optimization model is established
min L ∈ R r × q 1 : L T L = I q 1 P ∈ R c × q 2 : P T P = I q 2 M i ∈ R q 1 × q 2 , i = 1 , ⋯ , m ∑ i = 1 m ‖ X i − L M i P T ‖ F 2 (2)
Adopt iterative method, fix P first, use M L = ∑ i = 1 m X i P P T X i T get the q 1 largest eigenvalue pairs L ∈ R r × q 1 of M L . Then fix L, use M P = ∑ i = 1 m X i T L L T X i get the q 2 largest eigenvalue pairs P ∈ R c × q 2 of M L . And finally, solve M i through M i = L T X i P . Repeat the solution until the requirements are met. Then get new dimension reduction dataset T ′ = { ( X 1 , X 2 , ⋯ , X m ) } ∈ R q 1 × q 2 , i = 1 , ⋯ , m . In this paper, each sample after dimensionality reduction is pulled into column vectors to form sample matrix T r = { ( r i , y i ) } i = 1 m ∈ R d ′ × { ± 1 } similar to section 2.1.
The algorithm proposed in this chapter is based on article (13), mainly introduces two improved models: PCA-L1-CSVM and GLRAM-L1-CSVM. The first model uses data set is T = { ( x i , y i ) } i = 1 m ∈ R d × { ± 1 } . The second model uses data set is T = { ( X i , y i ) } i = 1 m ∈ R r × c × { ± 1 } . A = [ x 1 + , ⋯ , x m 1 + ] , B = [ x 1 − , ⋯ , x m 2 − ] , X = [ A , B ] represents the matrix of the positive sample, the matrix of the negative sample, and the matrix of the population sample. I 1 = { i ∈ { 1 , ⋯ , m 1 } : y i = 1 } and I 2 = { j ∈ { 1 , ⋯ , m 2 } : y j = − 1 } represents the subscript set of positive and negative data respectively.
PCA-L1-CSVM uses dimension reduction dataset T r = { ( r i , y i ) } i = 1 m ∈ R d ′ × { ± 1 } . The following planning model is constructed
min w , b , r i , v i , ξ i 1 2 ‖ w ‖ 2 + C ∑ i ∈ I ξ i s .t . y i ( w T r i + b ) ≥ 1 − ξ , ξ i ≥ 0 , i = 1 , ⋯ , m , min r i ∑ i = 1 m ‖ ∑ j = 1 d ′ r i j v j − x ˜ i ‖ 2 2 s .t . ‖ v i ‖ 2 = 1 , v i T v j = 0 ( i ≠ j ) . (3)
where C > 0 is a penalty parameter, the second condition of the constraint is to use PCA dimensionality reduction for the original data set. First, PCA is used to calculate the data after dimensionality reduction, and then the model is represented as an unconstrained model
min w , b 1 2 ‖ w ‖ 2 + C ∑ i = 1 m max ( 1 − y i ( w T r i + b ) , 0 ) + 1 (4)
Because bias has little effect on model performance, bias is often ignored in recent studies, and r i T ← [ r i T , 1 ] ∈ R d ′ + 1 , w T ← [ w T , b ] ∈ R d ′ + 1 for convenience, this section d ′ + 1 is denoted as d ′ . The model (4) was rewritten as an unconstrained quadratic programming model
min w f ( w ) : = 1 2 ‖ w ‖ 2 + C ∑ i = 1 m max ( 1 − y i ( w T r i ) , 0 ) (5)
Model (5) is expressed in matrix form
min w 1 2 ‖ w ‖ 2 + C max ‖ 0 , B w + d ‖ 1 (6)
where
B = − [ y 1 r 1 T ⋮ y m r m T ] ∈ R m × d ′ , d = [ 1 ⋮ 1 ] ∈ R d ′ .
Introducing variable s ∈ R d ′ , we obtain the following constrained optimization problem
min w 1 2 ‖ w ‖ 2 + P ( s ) s .t . s = B w + d , (7)
and P ( s ) = C max ‖ 0 , s ‖ 1 .
Consider the ALM function of model (7) above:
L σ ( w , s , λ ) = 1 2 ‖ w ‖ 2 + P ( s ) − λ T ( s − B w − d ) + σ 2 ‖ s − B w − d ‖ 2
At iteration k, solve
min w , s L σ k ( w k , s k , λ k ) , (8)
obtain ( w k + 1 , s k + 1 ) , and then update the Lagrange multiplier λ k + 1 .
PCA-L1-CSVM uses dimension reduction dataset T ′ = { ( X 1 , X 2 , ⋯ , X m ) } ∈ R q 1 × q 2 , i ∈ I . The following planning model is constructed
min w , b , r i , v i , ξ i 1 2 ‖ w ‖ 2 + C ∑ i ∈ I ξ i s .t . y i ( w T r i + b ) ≥ 1 − ξ , ξ i ≥ 0 , i = 1 , ⋯ , m , min r i ∑ i = 1 m ‖ X i − L M i , R P T ‖ F 2 s .t . L T L = I q 1 , P T P = I q 2 (9)
where C > 0 is a penalty parameter, different from the second constraint condition in formula (3), the second constraint of formula (9) is to use GLRAM to reduce the dimension of the original data. First, GLRAM is used to reduce the dimension of the data, and pull the data matrix into column vectors. Then, the model is transformed into an unconstrained optimization model as (4). Then, the following operations are solved using the same method as in Section 3.1.
In order to test the effectiveness of Algorithm 1 and GLRAM-L1-CSVM, a series of comparative experiments are conducted on MNIST and fashion-MNIST data in this section. Each dataset contains 10 image types, each of which has a pixel size of 28 × 28. Eight groups of data were randomly selected from each data set and divided into four groups for binary comparison test. This experiment only uses the training set data of each kind of data set and divides it into training set and test set. The size of the training set is 3000, 4000 and 5000 respectively, accounting for 50%, 66% and 83% of the total data set, and the rest is the test set. In the experiment, the eigenvalues of 81, 100 and 121 after dimensionality reduction were used. All the features in the table refer to the features of the improved algorithm.
The experimental platform used in this experiment is matlab R2014B, and the Windows specification is Windows 10. The device specifications are as follows: The processor is Inter(R) Core (TM) I7-10700 (2.90 GHz), and the onboard RAM is 8 GB.
Algorithm 1. PCA-L1-CSVM.
A bar chart is used to represent the comparison of the accuracy of the first two groups of binary classification experiments of different algorithms, and a broken line chart is used to represent the running time of each algorithm in the first two groups of experiments. The bar charts and line charts from left to right of each image are the data results of dimensionality reduction to 81, 100, 121, respectively. The average results of each algorithm under the latter two groups of experiments are shown in the table. The data in bold is the best data, the time unit is second, and the precision unit is %.
The experimental results on the MNIST dataset are presented.
According to
Next, the experimental results on the Fashion-MNIST dataset are presented.
It can be seen from
| TN | Feature | L1-CSVM | PCA-L1-CSVM | GLRAM-L1-CSVM |
|---|---|---|---|---|
| 3000 | 81 100 121 | 98.67 98.67 98.67 | 98.95 98.96 99.08 | 99.09 99.02 99.00 |
| 4000 | 81 100 121 | 98.91 98.91 98.91 | 99.02 98.98 99.05 | 99.00 99.11 99.13 |
| 5000 | 81 100 121 | 99.00 99.00 99.00 | 99.16 99.40 99.37 | 99.32 99.24 99.21 |
| TN | Feature | L1-CSVM | PCA-L1-CSVM | GLRAM-L1-CSVM |
|---|---|---|---|---|
| 3000 | 81 100 121 | 0.3212 0.3177 0.3088 | 0.0362 0.0397 0.0446 | 0.0295 0.0376 0.0448 |
| 4000 | 81 100 121 | 0.5138 0.4980 0.5114 | 0.0424 0.0496 0.0644 | 0.0401 0.0522 0.0641 |
| 5000 | 81 100 121 | 0.9036 0.9132 0.8660 | 0.0657 0.0729 0.0914 | 0.0544 0.0584 0.0906 |
| TN | Feature | L1-CSVM | PCA-L1-CSVM | GLRAM-L1-CSVM |
|---|---|---|---|---|
| 3000 | 81 100 121 | 99.27 99.27 99.27 | 99.59 99.54 99.44 | 99.80 99.62 99.63 |
| 4000 | 81 100 121 | 99.55 99.55 99.55 | 99.58 99.64 99.48 | 99.74 99.67 99.63 |
| 5000 | 81 100 121 | 99.50 99.50 99.50 | 99.65 99.55 99.65 | 99.78 99.73 99.78 |
| TN | Feature | L1-CSVM | PCA-L1-CSVM | GLRAM-L1-CSVM |
|---|---|---|---|---|
| 3000 | 81 100 121 | 1.4102 1.4857 1.5022 | 0.1978 0.1947 0.2272 | 0.2080 0.1926 0.2272 |
| 4000 | 81 100 121 | 1.9142 1.9132 2.1364 | 0.2294 0.2491 0.2964 | 0.2128 0.2306 0.3006 |
| 5000 | 81 100 121 | 2.4396 2.4949 2.4380 | 0.3044 0.3720 0.3917 | 0.3072 0.3294 0.3807 |
In this paper, the ALM is applied to solve two models: PCA-L1-CSVM and GLRA-L1-CSVM. The main contribution of this paper is as follows: Firstly, principal component analysis or GLRAM method is introduced to reduce dimension of data set. Then, the data after dimensionality reduction is dichotomized using L1-CSVM. Experimental results show that the improved model can greatly reduce the running time while maintaining a general increase in accuracy. In the future, we hope to use ALM to solve the original problems of other models.
Supported by Natural Science Foundation of Shandong Province, Funded Project Nos. ZR2020MA026, ZR2018BF010.
The authors declare no conflicts of interest regarding the publication of this paper.
Cui, M.Z. and Fan, L.Y. (2022) A New Algorithm for Reducing Dimensionality of L1-CSVM Use Augmented Lagrange Method. Journal of Applied Mathematics and Physics, 10, 21-30. https://doi.org/10.4236/jamp.2022.101003