Consider a random sample $\left(X_1,Y_1\right),\cdots ,\left(X_n,Y_n\right)$ of iid $\left(p+1\right)$-dimensional random vectors, where $Y_i$ is univariate and $X_i={\left(X_{1i},\cdots ,X_{pi}\right)}^{\text{T}}$ is a p-dimensional, $i=1,\cdots ,n$. Let $m\left(X\right)=E\left(Y|X\right)$ and write

$Y=m\left(X\right)+\epsilon ,$(1)

where $\epsilon =Y-m\left(X\right)$. For $k=1,\cdots ,p$, we consider p marginal nonparametric regression functions

$m_k\left(x\right)=\text{E}\left(Y_i|X_{ki}=x\right)$(2)

of Y on each variable $X_k$, and define the set of active and inactive predictors by

$\mathcal{D}=\left\{k:m_k\left(x\right)\text{is}\text{not}\text{a}\text{constant}\text{function}\right\},{\mathcal{D}}^c=\left\{1,\cdots ,p\right\}-\mathcal{D},$(3)

respectively. The proposed screening procedure relies on ranking the significance of the p covariates according to the magnitude of the variance of their respective marginal regression functions,

${\sigma }_{m_k}^2=\text{var}\left(m_k\left(X\right)\right)\text{for}k=1,\cdots ,p.$(4)

Note that ${\sigma }_{m_k}^2>0$ for $k\in \mathcal{D}$, while ${\sigma }_{m_k}^2=0$ for $k\in {\mathcal{D}}^c$, making ${\sigma }_{m_k}^2$ a natural quantity to discriminate between the two classes of predictors. In addition, the variance of the regression function appears as the mean shift, under local alternatives, of the procedure for testing the significance of a covariate proposed in Wang, Akritas and Van Keilegom [8] . This suggests that ${\sigma }_{m_k}^2$ is the best quantity to discriminate between the two classes of predictors.

If ${\hat{m}}_k$ denotes an estimator of $m_k$, ${\sigma }_{m_k}^2$ can be estimated by the sample variance of ${\hat{m}}_k\left(X_{k1}\right),\cdots ,{\hat{m}}_k\left(X_{kn}\right)$. The methodology described here works with any type of nonparametric estimator of $m_k$, but the theory has been developed for Nadaraya-Watson type estimators.

For a kernel function $K(\cdot )$ and bandwidth h, set ${\hat{m}}_k\left(X_{ki}\right)={\displaystyle {\sum }_{j=1}^n{Y}_j{W}_{k;i,j}}$, where , and

${\tilde{S}}_{m_k}^2=\frac{1}{n}\underset{i=1}{\overset{n}{{\displaystyle \sum }}}{\left({\hat{m}}_k\left(X_{ki}\right)-\frac{1}{n}\underset{l=1}{\overset{n}{{\displaystyle \sum }}}{\hat{m}}_k\left(X_{kl}\right)\right)}^2$(5)

for the estimator of ${\sigma }_{m_k}^2$. The bandwidth will be of the order $h=c{n}^{-1/5}$, throughout this paper. The RV-SIS estimates $\mathcal{D}$ by

$\hat{\mathcal{D}}=\left\{k:{\tilde{S}}_{m_k}^2\ge {\hat{C}}_d,\text{for}1\le k\le p\right\}$(6)

for some threshold parameter ${\hat{C}}_d$. Thus, the RV-SIS procedure reduces the dimension of covariate vector from p to $\left|\hat{\mathcal{D}}\right|$, where $|\cdot |$ refers the cardinality of a set. The choice of $C_d$, which defines the RV-SIS procedure, is discussed below.

We adopt the idea of the soft thresholding rule by Zhu et al. [9] as a method for choosing the threshold parameter $C_d$. This method consists of randomly generating a vector $Z=\left(X_{p+1},\cdots ,X_{p+d}\right)$ of d auxiliary random variables from the uniform distribution between (0, 1), $X_{p+i}~\text{Unif}\left(0,\text{1}\right)$ for $i=1,\cdots ,d$, that are independent of both $X$ and $Y$. By design, the auxiliary variables are inactive predictors. The soft thresholding rule chooses the threshold parameter as

${\hat{C}}_d=\underset{j\in ℬ}{\text{max}}{\tilde{S}}_{m_j}^2,$(7)

where $ℬ=\left\{p+1,\cdots ,p+d\right\}$ denotes the set of indices of the d auxiliary variables.

Theorem 1 provides an upper bound on the probability of selecting inactive predictors from using the proposed soft thresholding rule provided the following exchangeability condition holds.

Exchangeability Condition: Let $k\in {\mathcal{D}}^c$ and $j\in ℬ$. Then, the probability that ${\tilde{S}}_{m_k}^2$ is greater than ${\tilde{S}}_{m_j}^2$ is equal to the probability that ${\tilde{S}}_{m_k}^2$ is less than ${\tilde{S}}_{m_j}^2$.

Theorem 1. Under the exchangeability condition, for any integer $r\in \left(0,p\right)$ we have

$P\left(\left|\hat{\mathcal{D}}\cap {\mathcal{D}}^c\right|\ge r\right)\le {\left(1-\frac{r}{p+d}\right)}^d.$(8)

For some constants $c>0$ and $0<\kappa <2/5$.

A practical issue using the soft thresholding is how to choose the size of auxiliary variable d. Numerical simulation results suggested that $d=p/2$ works well on simulated data.

The RV-SIS procedure consists of the following steps:

1. Calculate a sample variance of the nonparametric estimator ${\tilde{S}}_{m_k}^2$ of each covariate for $k=1,\cdots ,p$.

2. Construct d auxiliary random variables and compute a sample variance of the nonparametric estimator ${\tilde{S}}_{m_j}^2$ of each auxiliary variable for $j=1,\cdots ,d$.

3. Select predictors whose sample variance of the estimator is greater than the maximum sample variance of the auxiliary variables.

In this section, we show that the RV-SIS possesses the sure screening property. The sure screening property is fundamental to a feature screening procedure. This property ensures that all active predictors are selected in the screened submodel with probability 1 as the sample size increases. The following conditions are required for technical proofs:

(C1) There exists positive constants $t,C_1$ and $C_2$ such that,

(a) $\underset{1\le k\le p}{\text{max}}\text{E}\left\{\text{exp}\left(t\left|Y_j-m_k\left(X_{kj}\right)\right|\right)\right\}<C_1<\infty$,

(b) $\underset{1\le k\le p}{\max }\text{E}\left(\exp \left(t{\left(X_{ki}-X_{kj}\right)}^2\right)\right)<C_2<\infty$.

(C2) The kernel $K(\cdot )$ has bounded support, is symmetric, and is Lipschitz continuous, i.e., it satisfies, for some ${\Lambda }_1<\infty$ and for all $u,u^{\prime }\in ℝ$,

$\left|K\left(u\right)-K\left(u^{\prime }\right)\right|\le {\text{Λ}}_1\left|u-u^{\prime }\right|.$

(C3) If $f_k\left(x\right)$ denotes the marginal density of the kth predictor, we have

$\underset{x}{\sup }{\left|x\right|}^{s}E\left(\left|Y\right||X_k=x\right)f_k\left(x\right)\le B<\infty \text{for}\text{some}s\ge 1$

${\sup }_x{f}_k\left(x\right)<\infty$, $\underset{x}{\text{inf}}{f}_k\left(x\right)>0$, and $f_k\left(x\right)$ is uniformly continuous, for all $k=1,\cdots ,p$.

(C4) The conditional expected value $m_k(\cdot )$ is a Lipschitz continuous for all $k=1,\cdots ,p$, that is for some ${\Lambda }_2<\infty$ and for all $u,u^{\prime }\in ℝ$,

$\left|m_k\left(u\right)-m_k\left(u^{\prime }\right)\right|\le {\text{Λ}}_2\left|u-u^{\prime }\right|.$

(C5) For some constants $c>0$ and $0<\kappa <2/5$, we have

$\underset{k\in \mathcal{D}}{\text{min}}{\sigma }_{m_k}^2\ge c{n}^{-\kappa }+C_d,$

where $C_d$ is ${\max }_{j\in ℬ}{\sigma }_{m_k}^2$.

In words, Condition (C1) requires that the moment generating functions of the absolute value of the error terms of the marginal regressions and the square difference between two covariates, is finite at least for some $t>0$. Conditions (C2) and (C3) are standard conditions for establishing uniform convergence rates of needed for the kernel density estimator. Condition (C5) sets a lower bound on the variance of the marginal regression functions of the active predictors.

Theorem 2. Let ${\sigma }_{m_k}^2,{\tilde{S}}_{m_k}^2,\mathcal{D},\hat{\mathcal{D}}$, be defined in (4), (5), (3) and (6), respectively.

1. Under condition (C1)~(C4) for any $0<\kappa <2/5$ and $0<\gamma <2/5-\kappa$, there exists positive constants $c,c_1$, and $c_2$ such that,

$\begin{array}{l}P\left(\underset{1\le k\le p}{\max }\left|{\tilde{S}}_{m_k}^2-{\sigma }_{m_k}^2\right|\ge c{n}^{-\kappa }\right)\\ \le O\left(p\left[n\exp \left(-c_1{n}^{4/5-2\left(\gamma +\kappa \right)}\right)+n^2\exp \left(-c_2{n}^{\gamma }\right)\right]\right)\end{array}$

2. Under condition (C1) ~ (C5), $c,c_1,c_2,\gamma$ and $\kappa$ as in part 1,

$P\left(\mathcal{D}\subseteq \hat{\mathcal{D}}\right)\ge 1-O\left(\left|\mathcal{D}\right|\left[n\exp \left(-c_1{n}^{4/5-2\left(\gamma +\kappa \right)}\right)+n^2\exp \left(-c_2{n}^{\gamma }\right)\right]\right),$

where $\left|\mathcal{D}\right|$ is the cardinality of $\mathcal{D}$.

The second part of Theorem 2 shows that the screened submodel includes all active predictors with the probability approaches to 1 with an exponential rate.

Here we present the result of several simulation studies comparing performance of the SIS, DC-SIS, NIS, RRCS and RV-SIS methods. In all cases,

$X={\left(X_1,X_2,\cdots ,X_p\right)}^{\text{T}}$ comes from a multivariate normal with mean zero and covariance $\Sigma ={\left({\sigma }_{ij}\right)}_{p\times p}$, and $\epsilon ~N\left(0,1\right)$. We use three different covariance matrices: (i) ${\sigma }_{ij}={0.5}^{\left|i-j\right|}$, (ii) ${\sigma }_{ij}={0.8}^{\left|i-j\right|}$, and (iii) ${\sigma }_{ij}=0.5$. We set the dimension of covariates p to be 2000 and the sample size n to be 200. We replicate the experiment 500 times and base the comparisons on the following three criteria.

R1: The 5%, 25%, 50%, 75%, 95% quantiles of the minimum model size that includes all active covariates.

R2: The proportion of times each individual active covariate is selected in models of size $d_1=\left[n/\log n\right]$, $d_2=\left[2n/\log n\right]$ and $d_3=\left[3n/\log n\right]$.

R3: The proportion of times all active covariates are selected in models of size $d_1=\left[n/\log n\right]$, $d_2=\left[2n/\log n\right]$ and $d_3=\left[3n/\log n\right]$.

Criterion R1 shows the performance of the ranking of the predictors of the different screening procedures. Criterion R2 and R3 shows the accuracy of the different screening procedure if we used the thresholding value suggested by Fan and Lv [2] .

To compare the performance of the screening procedure for both linear and nonlinear cases, we used the following four models:

(a) $Y=2X_1+0.5X_2+3\cdot 1\left\{X_{12}<0\right\}+2X_{22}+\epsilon$

(b) $Y=1.5X_1\cdot X_2+3\cdot 1\left\{X_{12}<0\right\}+2X_{22}+\epsilon$

(c) $Y=2\text{cos}\left(2\pi X_1\right)+0.5X_2^2+3\cdot 1\left\{X_{12}<0\right\}+2X_{22}+\epsilon$

(d) $Y=2\text{cos}\left(2\pi X_1\right)X_2^2+3X_{12}+2\text{exp}\left(1\left\{X_{22}<0\right\}\right)+\epsilon$

All models include an indicator variable. Model (a) is linear, model (b) includes an interaction of two active predictors, model(c) is additive but nonlinear, and model(d) is nonlinear with an interaction term.

Tables 1 - 3 present the simulation results for R1 using each of the above models with ${\sigma }_{ij}={0.5}^{\left|i-j\right|}$, ${\sigma }_{ij}={0.8}^{\left|i-j\right|}$, and ${\sigma }_{ij}=0.5$ respectively. Tables 4 - 6 presents the simulation results for R2 and R3 with ${\sigma }_{ij}={0.5}^{\left|i-j\right|}$, ${\sigma }_{ij}={0.8}^{\left|i-j\right|}$, and ${\sigma }_{ij}=0.5$ respectively.

These results show that the comparisons in term of the three criteria are similar. All procedures perform worse when we use the equal covariance matrix, ${\sigma }_{ij}=0.5$. SIS and RRCS perform rather poorly except in Model (a) where all methods have similar performance. For Model (b), NIS performs slightly better than RV-SIS, while RV-SIS performs somewhat better than DC-SIS when ${\sigma }_{ij}={0.8}^{\left|i-j\right|}$, considerably better when ${\sigma }_{ij}={0.5}^{\left|i-j\right|}$, and significantly better when ${\sigma }_{ij}=0.5$. In Models(c) and (d) DC-SIS and NIS have similar performance but RV-SIS performs significantly better than either of them.

Finally, Table 7 presents the execution time, in seconds, of the DC-SIS, NIS and RV-SIS for Model (d). The RV-SIS procedure takes significantly less time than the DC-SIS and slightly less time than the NIS.

In this section, we use simulations to compare the soft thresholding rule to the hard thresholding approaches for selecting the submodel. We consider following three models relating the response Y to covariates $X_1,X_2,\cdots ,X_p$, where $p=2000$:

(e) $Y=c_1{X}_1+\cdots +c_{25}{X}_{25}+\epsilon$

(f) $Y=c_1{X}_1+\cdots +c_{10}{X}_{10}+\epsilon$

(g) $Y=c_1{X}_1+c_2{X}_2+c_3{X}_3+c_4{X}_4+c_5{X}_5+\epsilon$,

where the covariate vector has the p-variate normal distribution with mean zero and covariance $\Sigma ={\left({0.5}^{\left|i-j\right|}\right)}_{p\times p}$, $\epsilon ~N\left(0,1\right)$, and the coefficients $c_1,\cdots ,c_{25}$ were randomly generated from the uniform distribution between (1, 2.5), and kept fixed throughout the simulation. From each of these models, we generated 500 data sets of size $n=200$.

For the soft thresholding approach, we randomly generate the auxiliary variable $Z=\left(X_{2001},\cdots ,X_{3000}\right)$, where the $X_{p+i}$ are independent Unif (0, 1). For the hard thresholding we consider three model sizes: $d_1=\left[n/\log n\right]=37$, $d_2=\left[2n/\log n\right]=75$, $d_3=\left[3n/\log n\right]=113$. The two approaches are compared in terms of the proportion of each active covariate is selected. We also record the 5%, 25%, 50%, 75% and 95% quantiles of the submodel size using the soft thresholding rule.

The 5%, 25%, 50%, 75% and 95% quantiles of the submodel size using the soft thresholding rule for Models (e), (f) and (g), are presented in Table 9 . The proportion that each of the active covariates is selected with the different approaches for Models (e), (f) and (g) are shown in Tables 6 - 8 , respectively.

From Table 9 , it is seen that all percentiles decrease as the number of active covariates decreases; this is a nice feature of the soft thresholding approach. Also, for all models, the median submodel size falls between d₁ and d₂, but is always closer to d₁. Regarding the proportion that each active predictor is included in the submodel, Table 6 and Table 7 show that soft thresholding outperforms hard thresholding with d₁ in Model (e), but does slightly worse in Model(f); hard thresholding with d₂ and d₃ outperform soft thresholding. Finally, Table 8 shows that all active predictors were selected 100% of the time by all approaches.