Due to the simplicity and interpretability, linear regression models play a significant role in modern statistical methods. The linear regression model aspires to find the linear relationship between the dependent variable and the non-stochastic explanatory variables for the prediction purpose.

Let us consider the linear regression model

where is the vector of observations on the dependent variable, is the matrix of observations on the non-stochastic predictor variables, is a vectors of unknown coefficients, and is the vector of random error terms, which is independent and identically normally distributed with mean zero and common variance, that is and.

It is well known that the Ordinary Least Square Estimator (OLSE) is the Best Linear Unbiased Estimator (BLUE) for finding the unknown parameter vector in model (1), which can be obtained by minimizing Error Sum of Squares (ESS),

with respect to, and defined as

However, the OLSE is unstable and produces parameter estimates with high variance when multicollinearity exists on. As a curative action to the multicollinearity problem, the biased estimators have been used by many researchers. The following biased estimators are popular in statistical literature:

• Principal Component Regression Estimator (PCRE) [1]

where is the first h columns of the standardized eigenvectors of.

• Ridge Estimator (RE) [2]

where is the regularization parameter, and is the identity matrix.

• r-k class estimator [3]

Note that r-k class estimator is a combination of PCRE and RE.

• Almost Unbiased Ridge Estimator (AURE) [4]

• Liu Estimator (LE) [5]

where is the regularization parameter.

• Almost Unbiased Liu Estimator (AULE) [6]

• r-d class estimator [7]

Note that r-d class estimator is a combination of PCRE and LE.

According to Kayanan and Wijekoon [8], the generalized form to represent the estimators RE, AURE, LE, AULE, PCR, r-k class estimator and r-d class estimator is given by

where

In recent studies, Kayanan and Wijekoon [8] have shown that r-k class estimator and r-d class estimator outperformed other estimators for the selected range of regularization parameter values when multicollinearity exists among the predictor variables. However, biased estimators introduce heavy bias when the number of predictor variables is high, and the final model may contain some irrelevant predictor variables as well. To handle this issue, Tibshirani [9] proposed the Least Absolute Shrinkage and Selection Operator (LASSO) as

where is a turning parameter. Note that we cannot find an analytic solution for LASSO since is a non-differentiable function. Tibshirani [9] and Fu [10] have used the standard quadratic programming technique and shooting algorithm, respectively, to find solutions for LASSO. Apart from these two methods, the Least Angle Regression (LARS) algorithm proposed by Efron et al. [11] is a popular one in the recent literature to find LASSO solutions. The LASSO wields both the multicollinearity problem and variable selection simultaneously in the linear regression model. However, LASSO failed to outperform RE if high multicollinearity exists among predictors, and it is unsteady when the number of predictors is higher than the number of observations [12]. To overcome this problem, Zou and Hastie [12] proposed Elastic net (ENet) estimator by combining LASSO and RE as

The LARS-EN algorithm, which is a modified version of the LARS algorithm, has been used to obtain solutions for ENet.

In this work, we propose generalized version of LARS algorithm that can be combined LASSO with other biased estimators such as AURE, LE, AULE, PCRE, r-k class and r-d class estimators. Further, we compared the prediction performance of proposed algorithm with existing algorithms of LASSO and Enet using a Monte-Carlo simulation study and real-world examples. The structure of the rest of the article is as follows: Section 2 contains proposed algorithms, Section 3 shows the comparison of proposed algorithm, Section 4 concludes the article, and references are provided at the end of the paper.

Based on Efron et al. [11] and Hettigoda [13], now we propose GLARS algorithm as follows:

Step 1: Standardize the predictor variables to have a mean of zero and a standard deviation of one, and response variable to have a mean zero.

Step 2: Start with the initial estimated value of as, and the residual.

Step 3: Find the predictor most correlated with as follows

Then increase the estimate of from 0 until any other predictor has a high correlation with the current residual as does. At this point, GLARS proceeds in the equiangular direction between the two predictors and instead of continuing in the direction based on.

In a similar way, the i^th variable eventually earns its way into the active set, and then GLARS proceeds in the equiangular direction between. Continue adding variables to the active set in this way moving in the direction defined by the least angle direction. In the intermediate steps, the coefficient estimates are updating using the following formula:

where is a value between [0,1] which represents how far the estimate moves in the direction before another variable enters the model and the direction changes again, and is the equiangular vector.

The direction is calculated using the following formula:

where is the matrix with column, be the j^th standard unit vector in which has the index of variables selected in each subsequent steps, and is a generalized variable which can be substituted by respective expressions for any of estimators of our interest as listed in Table 1. Then, is calculated as follows:

where

for any j such that, and

for any j such that.

Step 4: If, then is the matrix formed by removing the column from. Then the residual related to the current step is calculated as

and then move to the next step where is the value of j such that or or.

Step 5: Proceed Step 3 until.

In Table 1, is the identity matrix, is the number of selected variables in each subsequent step, and is the first columns of the standardized eigenvectors of.

Proposed algorithms are compared with the LARS-LASSO and LARS-EN algorithms using the RMSE criterion, which is the expected prediction error of the algorithms, and is defined as

where denotes the new data which are not used to obtain the parameter estimates, and is the estimated value of using the respective algorithm. A Monte Carlo simulation study and the real-world examples are used for the comparison.

This study clearly showed that the proposed LARS-rk and LARS-rd algorithms performed well in the high dimensional linear regression model when moderated and high multicollinearity existed among the predictor variables, respectively.

The appropriate algorithm for a particular practical problem can be chosen based on the variables of interest and prediction performance by referring to the plot of coefficient paths.

We thank the Postgraduate Institute of Science, University of Peradeniya, Sri Lanka for providing all facilities to do this research, and also we thank the reviewers for their valuable comments.

The authors declare no conflicts of interest regarding the publication of this paper.

Kayanan, M. and Wijekoon, P. (2020) Variable Selection via Biased Estimators in the Linear Regression Model. Open Journal of Statistics, 10, 113-126. https://doi.org/10.4236/ojs.2020.101009