[11] proposed an exact path-following algorithm leading to the QR solution (3). For a fixed value of τ , they approximate ρ τ ( r i ) by an asymmetric “bent arc” expressed as a piecewise polynomial denoted by S i , τ ( β ; c ) , where c is a small but known positive constant. Specifically, let A ( c ) = { i : − c τ < r i < c ( 1 − τ ) } , i = 1 , ⋯ , n , be a partition of the sample units. Then the approximation S i , τ ( β ; c ) is defined as follows

where ω i , u ( β ; c ) , u = 1 , 2 , are weights given by

and where ν i , g , g = 1 , 2 , 3 , 4 , are the following indicator functions

Note that we suppressed the dependence of ω i , u and ν i , g on τ . This simplification in notation also holds in the rest of the paper.

The smooth objective function takes the form

where ∑ i I ( i ∈ A ( c ) ) = | A ( c ) | , i.e., the cardinality of the set A ( c ) . Now for minimizing Equation (6), the idea is to define a piecewise linear solution using a decreasing sequence of values, say 0 < c J < ⋯ < c 1 , such that A ( c j + 1 ) ⊆ A j and | A ( c J ) | = p . Additionally, let c j > 0 be a fixed value and β ( c j ) the corresponding minimizer of (6), with c j chosen such that | A ( c j ) | > p . Given this setup, the set of constant weight indicator functions is given by D ( c j ) = { ( β ; c ) ∈ ℝ p × ℝ + | ∀ i , g : ν i , g ( β ; c ) = ν i , g ( c j ) } . Then the p -dimensional gradient vector of S τ ( β ; c ) can be written as follows

where W u ( c j ) = diag ( ω 1 , u ( c j ) , ⋯ , ω n , u ( c j ) ) , j = 1 , ⋯ , J , u = 1 , 2 . Here Y = ( Y 1 , ⋯ , Y n ) T and 1 n is an n -dimensional vector of ones. Solving ∂ S τ ( β ; c ) / ∂ β = 0 p gives the p -dimensional weighted least squares estimator

where

For every new value of c j , information from the previous solution is used. Then, it is not hard to show that

where r ^ i , Y ( c j ) = Y i − X i T β ^ Y ( c j ) is an estimate of the random error r i , Y ( τ ) = Y i − X i T β 0 ( τ ) . Thus if | A ( c j + 1 ) | = p , then β ^ τ ( c ) = X − 1 ( h ) Y ( h ) , with h ≡ | A ( c J ) | , is an iteratively weighted least squares (IWLS) estimator of β 0 ( τ ) .

Suppose that we have x = ( x 1 , x 2 , ⋯ ) T with sign vector sign ( x ) = ( sign ( x 1 ) , sign ( x 2 ) , ⋯ ) T . The proposed path-following algorithm is based on the following theorem.

Theorem1. ([11]) Let c J > 0 be a fixed value such that | A ( c J ) | = p . Then under the assumption that for any c ∈ [ 0 , c J ] , sign ( r ℓ ( β ^ τ ( c ) ) ) = sign ( r ℓ ( β ^ τ ( c J ) ) ) , ∀ ℓ ∈ 1 − | A ( c J ) | , the solution curve evaluated at 0 p is equal to the solution of (2), namely β ^ τ ( 0 ) = β ^ ( τ ) .

It is well known that conditional QR estimates are severely affected by the so-called curse-of-dimensionality when p is large. To tackle this issue, several authors have proposed dimension reduction methods. The general framework for these methods can be characterized by a set of nonzero coefficients A = { j : β j , 0 ( τ ) ≠ 0 , j = 1 , ⋯ , p } , and q = | A | is the cardinality of A . The set A is unknown and can be estimated by various (non)parametric approaches such as kernel-based smoothing or B-spline basis functions. Suppose that the true parameter vector β 0 ( τ ) is sparse. Hence, we can write β 0 ( τ ) = ( β 1 , 0 T ( τ ) , β 2 , 0 T ( τ ) ) T , where β 1 , 0 ( τ ) is a q × 1 vector and β 2 , 0 ( τ ) = 0 p − q . Moreover, it is unknown which coefficients are nonzero and which are zero. To this end, one can use a penalty function p λ ( ⋅ ) with tuning parameter λ which depends on n , and where for ease of presentation, we set λ ≡ λ n .

In this paper, we consider the SCAD penalty which is defined on ℝ + by its first derivatives as

where a > 2 and λ > 0 are tuning parameters, and where ( x ) + = max ( x , 0 ) . Note that (11) is symmetric, non-convex on [ 0 , ∞ ) , and singular at the origin. Its derivative vanishes outside [ − a λ , a λ ] . Following the recommendation of [5], we use a = 3.7 in the numerical applications. The parameter λ controls the complexity of the selected model, and λ → 0 as n → ∞ . The function (11) overcomes some of the undesirable properties of the LASSO penalty, where p λ ( x ) = λ | x | .

For a fixed c , let β ^ j , τ ( c ) , j = 1 , ⋯ , p , denote the univariate conditional (marginal) quantile estimator of β j , 0 ( τ ) obtained through the IWLS method of Section 3.1. Then, for given values of λ and τ , the proposed final one-step conditional QR estimator β ^ λ , τ ( c ) of β 0 ( τ ) can be obtained by minimizing the following penalized objective function

Thus, we use β ^ j , τ ( c ) as an initial estimate to determine the penalty weights for individual coefficients in QR. This adaptive approach is a natural extension of the local linear approximation as proposed by [23] and briefly discussed in Section 1. The approach allows the procedure to apply smaller penalties to coefficients with strong evidence of being non-zero, thereby improving selection accuracy⁴.

Let | A ^ λ , τ | be the number of nonzero estimates. Then, following [14], we choose the value of λ that minimizes the following high-dimensional Bayesian-type information criterion (BIC):

In the case β ^ j , τ ( c ) in (12) is replaced by an initial estimator not depending on c , simulation results by [9] show that if the choice of λ is based on (13), all relevant components are selected consistently as n → ∞ . Further, similar to Theorem 1, it follows that β ^ λ , τ ( c ) is equal in sign to the true β 0 ( τ ) (denoted by β ^ λ , τ ( c ) = s β 0 ( τ ) ) if and only if sign ( β ^ λ , τ ( c ) ) = sign ( β 0 ( τ ) ) .

Due to the non-smoothness and non-convexity of the SCAD penalty function (11), the classical Karush-Kuhn-Tucker optimality condition is not applicable to study the asymptotic properties of the final estimator β ^ λ , τ ( c ) . One common used technique to get around it in QR is to decompose a non-convex objective function as the difference of two convex (check loss) functions; see, e.g., [19] and [14]. Using this technique to decompose the SCAD function, and assuming that certain regularity conditions hold, the following theorem can be proven.

Theorem2. (Consistency) Assume that p ′ λ ( ⋅ ) = λ p ′ ( ⋅ ) . Suppose p ′ ( ⋅ ) is a continuous function on ( 0 , ∞ ) and there is some k > 0 such that p ′ ( θ ) = O ( θ − k ) as θ → 0 + , and q 1 / 2 n − k → 0 . In addition, let p ′ λ ( ⋅ ) be the SCAD function and λ n k → ∞ , then we have ℙ ( β ^ λ , τ ( c ) = s β 0 ( τ ) ) → 1 .

The result follows from similar arguments as in the proof of Theorem 4.2(i) in [12] and, hence, has been omitted from the text. [20] study the consistency of LASSO variable selection in a high-dimensional linear regression setting.

Remark (Oracle property) Under certain regularity conditions on the design of the model, [2], Theorem 3.2, proved sparsity and asymptotic normality of the SCAD-penalized quantile estimator β ^ λ , τ ( c ) .

Following [11], we now present an algorithm for selecting c . Ideally, c should be small enough to get a good approximate solution to the objective function (12). However, for small values of c estimation will be more difficult as the objective function S τ ( β ; c ) approaches ℒ ( β ) having a step-function gradient. The proposed pseudo code is given by Algorithm 1. High values of P lead to high c * and then to better convergence performance but to more biased estimates; small values of P are likely to cause failures in the estimating algorithm. Throughout the next two sections, we set c start = 0.5 and P = 0.7 . These choices are recommended by [11]. In particular, these authors provide guidance on selecting c and p when n is large ( n ≥ 10000 ). Among other things, their results indicate that it is possible to reduce P (or equivalently c ) without affecting the convergence performance of the IWLS.

Algorithm 1.An adaptive algorithm for selecting c.

In this example, we simulate 100 datasets from the p -dimensional linear regression model

where ( n , p ) = ( 100 , 200 ) and ( 100 , 400 ) , and where ε i is an independent and identically (i.i.d.) distributed random variable. The nonzero parameters are given by β 1 = 3 , β 2 = 1.5 , β 5 = 2 , and the rest of the β j ’s are zero.

We generate the covariates X i from the N ( 0 p , Σ ) multivariate normal distribution, where Σ = ( σ u , v ) p × p with σ u , v = r | u − v | and r = 0.5 . For the distribution of the error term, we consider the following three scenarios: 1) ε i ~ i .i .d . N ( 0 , 1 ) ; 2) ε i ~ i .i .d . t 3 , i.e., the Student t distribution with 3 degrees of freedom; and 3) a contaminated N ( 0 , 1 ) distribution having 10% outliers from a Cauchy distribution, denoted by C₁₀- N . We show results for quantile levels τ = 0.25 , 0.50 , and 0.75. We focus on the one-step quantile regression method SCAD penalty (denoted by QS), and also consider the one-step quantile regression method with LASSO penalty (denoted by QL) as a “benchmark”⁵. As a second benchmark for comparison, we compute the Oracle estimator, which is the least squares estimator of the true model.

To assess the performance of both quantile regression methods, we adopt the following criteria: 1) NN: average number of nonzero covariates; 2) True Positives (TP): average number of covariates that are correctly included in the model; 3) False Positives (FP): average number of covariates that are incorrectly included in the model; 4) AE: average of the absolute value of the estimation error which is defined as ∑ j = 1 p n | β ^ j , τ ( c * ) − β j , 0 ( τ ) | .

Table 1 summarizes the simulation results. The variable selection performance of QS is quite descent for all values of τ and p n . Indeed, except for one case (QL with τ = 0.75 , p n = 400 , and ε i ~ i .i .d . N ( 0 , 1 ) ), all computed TP values are equal to the true active model size of 3, with standard errors equal to zero. Hence, to keep the paper clear and readable, all computed TP values have been omitted from Table 1. The QL method tends to have substantially higher AE values than QS at different quantile levels for almost all p n values and error distributions. In the case ε i ~ i .i .d . t 3 , the QS method is near the oracle method in terms of AE, but the performance of QL is relatively poor. Also, the advantage of the QS method over the QL method can be seen by its lower AE values when the distribution of ε i is heavy-tailed. In general, the proposed QS method tends to be efficient for the three error distributions under study.

Table 1. Variable selection performance of one-step quantile regression with LASSO (QL) and SCAD (QS) type penalties for n = 100 . Standard errors are in parentheses. NN = Average number of nonzero covariates; FP = False positives; AE = Average of the absolute values of the estimation error.

We next compare the execution time of the QL and QS methods with initial quantile parameter estimates obtained by two methods: 1) the IWLS method of Section 3.1, and by 2) the well-known quantile regression method rq in the R-quant package and denoted here by the short-hand notation “bet”. The simulation setting is the same as in Section 4.1 with Scenario 1. However, in this case n = 80 , n = 1000 , and p n = 100 . Using the R-package microbenchmark, Figure 1(a) and Figure 1(b) show plots of the microbenchmark timings for three different values of τ , with “neval” denoting the number of replicates.

(a)

(b)

Figure 1. Autoplots of microbenchmark timings: plot (a) is for ( n = 80 , p n = 100 ), and plot (b) is for ( n = 1000 , p n = 100 ).

Figure 1(a) shows that there is hardly any difference between quantile estimates obtained with the QS-IWLS method and the QL-bet method for most values of τ . However, we see from Figure 1(b) that the difference between both methods is more noteworthy for the case ( n , p n ) = ( 1000 , 100 ) . Indeed, QL is about 2 times faster than QS, for all values of τ . This difference in computing time was also noted for simulation results obtained with the Student t 3 error distribution and with the contaminated N ( 0 , 1 ) error distribution, as specified earlier in Section 4.1. Thus, in terms of computing time, there is a price to be paid for using the proposed one-step QS-IWLS estimation method when n is large.

To assess the performance of the one-step variable selection method, we randomly selected 200 observations from the complete dataset. We used these observations

Figure 2. Plots of the estimated additive functions for seven predictors, with associated standard errors (dotted lines), using the R-gam function.

for model fitting and variable selection. The remaining 114 observations are used for checking the prediction performance of the fitted model. This procedure is repeated 100 times through the bootstrapping technique. Given this setup, and to guard against false positives, we declare a covariate to be relevant when it is selected 60% of the time.

Panel I of Table 2 contains the variable selection results for QL and QS, respectively. For τ = 0.25 , QL identifies different covariates than QS. However, for τ = 0.5 both methods select the variables Age and Alcohol as relevant covariates. On the other hand, when τ = 0.75 we see that the variables Age and Smoke1 are selected frequently. In terms of median absolute prediction error (MAPE) computed across all covariates, we have MAPE (QS) = 0.27 and MAPE (QL) = 0.37 for τ = 0.25 . When τ = 0.5 , MAPE (QL) = MAPE (QS) = 0.36, and when τ = 0.75 the MAPEs are equal to 0.36 for both methods.

Table 2. Frequency (≥60%) of covariates selected by QL and QS at three different quantiles based on 100 bootstrap replicates. Selected frequencies are in parentheses.

To investigate the QL and QS methods in high dimensions, we add 200 irrelevant covariates to the dataset. These new variables are randomly drawn from a U (0, 1) distribution. Hence, the augmented dataset consists of 215 covariates (200 irrelevant and 15 relevant ones). Using the same procedure as above, Panel II of Table 2 summarizes the variable selection results. For τ = 0.25 and τ = 0.75 , no covariates are selected by QS while QL selects covariates at these quantiles. Clearly, QS tends to produce more sparse models than QL. Further, in terms of MAPE, the QS method performs better than QL at τ = 0.25 . Specifically, MAPE (QS) = 0.28 and MAPE (QL) = 0.37. For τ = 0.5 MAPE (QL) = MAPE (QS) = 0.36, while for τ = 0.75 MAPE (QL) = 0.35 and MAPE (QS) = 0.34.

Figure 3 presents a boxplot of the prediction errors of QL and QS at different quantiles using the augmented plasma dataset, and based on 100 bootstrapped replications. The QS method at τ = 0.25 provides a good prediction, followed by the same method at τ = 0.5 and τ = 0.75 . Overall, the one-step QS method is found to be a good variable selection procedure in detecting the significant variables, even when some irrelevant variables are included in the dataset under study. The prediction errors of the proposed method are also smaller than those using the one-step QL method.

Figure 3. Boxplot of the prediction errors for the augmented plasma dataset.