Ultrahigh-dimensional data are commonly available in a wide range of scientific research and applications. Feature screening plays an essential role in the ultrahigh-dimensional data, where Fan and Lv [1] first proposed the sure independence screening (SIS) in the seminal paper. For linear regressions, they showed that the approach based on Pearson correlation learning possesses a sure screening property. That is, even if the number of predictors p can grow much faster than the number of observations n with log p = O ( n α ) for some

α ∈ ( 0, 1 2 ) , all relevant predictors can be selected with probability tending to one [2].

Lots of feature screening is the Model-based and Model-free approaches have been developed in recent years, see, for example, Wang [3] proposed forward regression for ultrahigh-dimensional data. Fan and Song [4] applied the maximum marginal likelihood estimates or the maximum marginal likelihood to ultrahigh-dimensional screening in generalized linear model. Fan et al. [5] further extend the correlation learning to marginal nonparametric learning. Zhu et al. [6] proposed a model-free feature screening approach for ultrahigh-dimensional data. Li et al. [7] proposed a robust rank correlation screening method to deal with ultrahigh-dimensional data based on the Kendall τ correlation coefficient. Li et al. [8] applied the distance correlation to sure independence screening procedure. He et al. [9] proposed a quantile-adaptive framework for nonlinear variable screening with high-dimensional heterogeneous data. Fan et al. [10] proposed nonparametric independence screening selects variables by ranking a measure of the nonparametric marginal contributions of each covariate given the exposure variable. Liu et al. [11] proposed a feature screening procedure for varying coefficient model based on conditional correlation coefficient. Nandy et al. [12] proposed a covariate information number sure independence screening, which used a marginal utility connected to the notion of the traditional Fisher information. Pouyap et al. [13] proposed a merge of the features selection methods in order to define the most relevant features in the texture of the vibration signal images.

To address the ultrahigh-dimensional feature screening in classification problem, Fan and Fan [14] proposed the t-test statistic for two-sample mean problem as a marginal utility for feature screening and establish its theoretical properties. Mai and Zou [15] applied the Kolmogorov filter to ultrahigh-dimensional binary classification. Cui et al. [16] proposed a screening procedure via used empirical conditional distribution functions. Lai et al. [17] proposed a feature screening procedure based on the expected conditional Kolmogorov filter for binary classification problem. However, the above-proposed screening methods assume that the types of data are continuous. For categorical covariates, Huang et al. [18] constructed a model-free discrete feature screening method based on the Pearson Chi-square statistics and showed its sure screening property fulfilling (Fan et al. [2]). When all the covariates are binary, Ni and Fang [19] proposed a model-free feature screening procedure based on information entropy theory for multi-class classification. Ni et al. [20] further proposed a feature screening procedure based on weighting Adjusted Pearson Chi-square for multi-class classification. Sheng and Wang [21] proposed a new model-free feature screening method based on classification accuracy of marginal classifiers for ultrahigh-dimensional classification. Anzarmou et al. [22] proposed a new model-free interaction screening method, termed Kendall Interaction Filter (KIF), for the classification in high-dimensional settings.

Based on the above study of classification models, in this paper, we propose a model-free feature screening for ultrahigh-dimensional multi-classification with both categorical and continuous covariates. The proposed feature screening method will be based on Gini impurity to evaluate the prediction power of covariates. Gini impurity is a non-purity attribute splitting index, which was proposed by Breiman et al. [23] and has been widely used in decision tree algorithms such as CART and SPRINT. With regard to categorical covariate screening, we can apply the index of purity gain, which is the same as information gain [19]. Similar to Ni and Fang [19], continuous covariates can be sliced via standard normal quantile. The proposed feature screening procedure is based on purity gain, which is referred to Purity Gain sure independence screening (PG-SIS). Theoretically, the PG-SIS is rigorously proven to enjoy. Fan and Lv [1] proposed sure screening property that ensures all important features can be obtained. Practically, as shown by the simulation results, compared with the existing feature screening method, PG-SIS satisfies the sure screening property.

This paper is organized as follows. Section 2 describes the proposed PG-SIS method in detail. Section 3 establishes its sure screening property. In Section 4, numerical simulations and an example for real data analysis are given to assess sure screening property of our method. Some concluding remarks are given in Section 5 and all the proofs are given in the Appendix.

We first introduce Gini impurity and purity gain, and then propose the screening procedure based on purity gain.

In this section, we establish the sure screening property of PG-SIS. Based on Ni and Fang [19] proposed sure independence screening theories, the following conditions are assumed.

Condition 1 (C1). There exist two positive constants c 1 and c 2 such that, c 1 / R ≤ p r ≤ c 2 / R , c 1 + c 2 ≤ R , c 1 / R ≤ p k , j r ≤ c 2 / R and c 1 / J k ≤ w k , j ≤ c 2 / J k for every 1 ≤ j ≤ J k , 1 ≤ r ≤ R and 1 ≤ k ≤ p .

Condition 2 (C2). There exist a positive constant c > 0 and 0 ≤ τ < 1 / 2 such that min k ∈ D e k ≥ 2 c n − τ .

Condition 3 (C3). R = O ( n ε ) , J = max 1 ≤ k ≤ P J k = O ( n k ) , where ε ≥ 0 , κ ≥ 0 and 2 τ + 2 ε + 2 κ < 1 .

Condition 4 (C4). There exist a positive constant c 3 , such that 0 < f k ( x | Y = r ) < c 3 for any 1 ≤ r ≤ R , and x in the domain of X k , where f k ( x | Y = r ) is the Lebesgue density function of X k conditional on Y = r .

Condition 5 (C5). There exist a positive constant c 4 and 0 ≤ ρ < 1 / 2 such that f k ( x ) ≥ c 4 n − ρ for any 1 ≤ k ≤ p and x in the domain of X k , where f k ( x ) is the Lebesgue density function of X k . Furthermore, f k ( x ) is continuous in the domain of X k .

Condition 6 (C6). R = O ( n ε ) , J = max 1 ≤ k ≤ P J k = O ( n κ ) , where 2 τ + 2 ε + 2 κ + 2 ρ < 1 and ε ≥ 0, κ ≥ 0 .

Condition 7 (C7). lim inf p → ∞ { min k ∈ D e k − max k ∈ I e k } ≥ δ , where δ > 0 is a constant.

Condition (C1) guarantees that the proportion of each class of variables cannot be either extremely small or extremely large. Similar assumption is also made in condition (C1) in Huang et al. [18] and Cui et al. [16]. According to Fan and Lv [1] and Cui et al. [16], Condition (C2) allows the minimum true signal to disappear to zero in the order of n − τ as the sample size goes to infinity. According to [19] Condition (C3) provides the covariates to diverge with a certain order and the number of classes for the response, and Condition (C6) modifies Condition (C3) a little bit. To ensures the sample percentiles are close to the true percentiles, Condition (C4) rules out the extreme case that some X k put heavy mass in a small range. Condition (C5) asks for the n − ρ as lower bound to the density. According to [16] and Zhu et al. [6] proposed ranking consistency property, we need to assume the inactive covariate subset I = { 1, ⋯ , p } \ D , then Condition (C7) is established.

Theorem 3.1. (Sure screening property) Under conditions (C1) to (C3), if all the covariates are categorical, we get:

P ( D ⊆ D ^ ) ≥ 1 − O ( p exp { − b n 1 − ( 2 τ + 2 ε + 2 κ ) + ( ε + κ ) log n } )

Theorem 3.2. (Sure screening property) Under conditions (C4) to (C6), when the covariates are composed of continuous and categorical variables, we get:

P ( D ⊆ D ^ ) ≥ 1 − O ( p exp { − b n 1 − ( 2 τ + 2 ε + 2 κ + 2 ρ ) + ( ε + κ ) log n } )

where b is a positive constant. If log p = O ( n α ) and α < 1 − ( 2 τ + 2 ε + 2 κ + 2 ρ ) , PG-SIS has a sure screening property.

Theorem 3.3. (Ranking consistency property) Under conditions (C1), (C4), (C5) and (C7), if log R J log n = O ( 1 ) and max { log P , log n } R 4 J 4 n 1 − 2 ρ = o ( 1 ) , then lim inf n → ∞ { min k ∈ D e ^ k − max k ∈ I e ^ k } > 0 , a.s.

Theorem 3.3 testifies that the proposed screening index can separate active and inactive covariates well in the sample level.

In the data, there are continuous and categorical covariates, and the response is categorical, which is very common in practice, but the applicable screening methods are very limited. We propose a PG-SIS procedure based on Gini impurity to effectively screen covariates. PG-SIS has a sure screening property and ranking consistency property theoretically and is model free. When the numbers of categories of all the covariates are the same and different, PG-SIS is quite similar to IG-SIS in performance, which can be shown in simulation.

The features screening reports some difficulties based on missing data. In the future, based on the classification model, we intend to propose a new feature screening to either the missing variable or the response variable can be selected.

The work was supported by National Natural Science Foundation of China [grant number 71963008].

The authors declare no conflicts of interest regarding the publication of this paper.

Wang, Z.Z. and Deng, G.M. (2022) Model-Free Feature Screening Based on Gini Impurity for Ultrahigh-Dimensional Multiclass Classification. Open Journal of Statistics, 12, 711-732. https://doi.org/10.4236/ojs.2022.125042

Proof of Proposition 2.1. To prove Proposition 2.1, we need to define f ( x ) = x 2 , proved to be close Ni and Fang [19]. By Jensen’s inequality [24],

∑ j = 1 J k     w k , j ∑ r = 1 R     p k , j r 2 = ∑ r = 1 R [ ∑ j = 1 J k     w k , j f ( p k , j r ) ] ≥ ∑ r = 1 R     f ( ∑ j = 1 J k     w k , j p k , j r ) = ∑ r = 1 R     f ( ∑ j = 1 J k     P ( X k = j ) P ( Y = r | X k = j ) ) = ∑ r = 1 R     p r 2

then

P G = 1 − ∑ r = 1 R     p r 2 − ∑ j = 1 J k     w k , j ( 1 − ∑ r = 1 R     p k , j r 2 ) = 1 − ∑ r = 1 R     p r 2 − ∑ j = 1 J k     w k , j + ∑ j = 1 J k     w k , j ∑ r = 1 R     p k , j r 2 = ∑ j = 1 J k     w k , j ∑ r = 1 R     p k , j r 2 − ∑ r = 1 R     p r 2 ≥ 0

The above equation holds if and only if p k , j r = p k , j ′ r , for any 1 ≤ r ≤ R and 1 ≤ j ≤ j ′ ≤ J . That is, X k and Y are independent.

Proof of Proposition 2.2. From the same proof as Proposition 2.1, we can get P G J ( Y | X k ) ≥ 0 holds if and only if p k , j ′ r = p k , j ′ r for any 1 ≤ r ≤ R and 1 ≤ j ≤ j ′ ≤ J , that is P ( Y = r | X ∈ ( q ( j − 1 ) , q ( j ) ] ) = P ( Y = r | X ∈ ( q ( j ′ − 1 ) , q ( j ′ ) ] ) . So when X k and Y are independent, P G J ( Y | X k ) = 0 . On the other hand, when P G J ( Y | X k ) = 0 for any J, we need to show that P ( X ≤ x | Y = r ) does not depend on r for any x in the domain of X. Proposition 2.2 has been proven by [19], so the proof is omitted here.

Lemma 1 (Bernstein inequality). If Z 1 , ⋯ , Z n is an independent random variable with a mean value of 0 and bounded supporter is [ − M , M ] , then the inequality:

P ( | ∑ i = 1 n     Z i | > t ) ≤ 2 exp { − t 2 2 ( v + M t 3 ) }

where v ≥ V a r ( ∑ i = 1 n     Z i )

Lemma 2. For discrete covariates X k and discrete response Y, we have the following three inequalities:

1) P ( | p ^ r − p r | > t ) ≤ 2 exp { − 6 n t 2 3 + 4 t }

2) P ( | w ^ k , j − w k , j | > t ) ≤ 2 exp { − 6 n t 2 3 + 4 t }

3) P ( | p ^ k , j r − p k , j r | > t ) ≤ 2 exp { − 6 n t 2 3 + 4 t }

Proof of Lemma 2. Three inequalities are similar in the proofs, where inequality (1) and inequality (2) have been given at [27]. The following is the proof of inequality (3).

p ^ k , j r = ∑ i = 1 n I { y i = r , X i , k = j } ∑ i = 1 n I { X i , k = j }

The expectation of p ^ k , j r is

E ( p ^ k , j r ) = E ( ∑ i = 1 n I { y i = r , X i , k = j } ∑ i = 1 n I { X i , k = j } ) = E ( ∑ i = 1 n y i = r , X i , k = j ∑ i = 1 n X i , k = j ) = p k , j r

Let Z i = I { y i = r | X i , k = j } − p k , j r , V a r ( ∑ i = 1 n     Z i ) = n V a r ( Z i ) = n p k , j r ( 1 − p k , j r ) ≤ n 4 is known, then

P ( | p ^ k , j r − p k , j r | > t ) = P ( | n − 1 ∑ i = 1 n     Z i | > t ) = P ( | ∑ i = 1 n     Z i | > n t ) ≤ 2 exp { − n 2 t 2 2 ( n 4 + n t 3 ) } ≤ 2 exp { − 6 n t 2 3 + 4 t }

According to Bernstein inequality, the formula is held.

Lemma 3. With regard to discrete covariates X k and discrete response Y, for any 0 < ε < 1 , under condition (C1), we have

P ( | e ^ k − e k | > 2 ε ) ≤ O ( R J 3 ) exp { − c 5 n ε 2 R 2 J 6 } ,

where c 5 represents a positive constant.

Proof of Lemma 3. By e k and e ^ k in Section 2.2, we have

log J K ( e ^ k − e k ) = [ ( 1 − ∑ r = 1 R     p ^ r 2 ) − ∑ j = 1 J k     w ^ k , j ( 1 − ∑ r = 1 R     p ^ k , j r 2 ) ] − [ 1 − ∑ r = 1 R     p r 2 − ∑ j = 1 J k     w k , j ( 1 − ∑ r = 1 R     p k , j r 2 ) ] = ( ∑ r = 1 R     p r 2 − ∑ r = 1 R     p ^ r 2 ) + ( ∑ j = 1 J k     w k , j − ∑ j = 1 J k     w ^ k , j ) + ( ∑ j = 1 J k     w ^ k , j ∑ r = 1 R     p ^ k , j r 2 − ∑ j = 1 J k     w k , j ∑ r = 1 R     p k , j r 2 ) = ∑ r = 1 R ( p r 2 − p ^ r 2 ) + ∑ j = 1 J k ( w k , j − w ^ k , j ) + ∑ j = 1 J k     ∑ r = 1 R ( w ^ k , j p ^ k , j r 2 − w k , j p k , j r 2 )

= ∑ r = 1 R ( p r − p ^ r ) ( p r + p ^ r ) + ∑ j = 1 J k ( w k , j − w ^ k , j )         + ∑ j = 1 J k     ∑ r = 1 R [ ( w ^ k , j p ^ k , j r + w k , j p k , j r ) ( p ^ k , j r − p k , j r ) + p ^ k , j r p k , j r ( w k , j − w ^ k , j ) ] = I 1 + I 2 + I 3

Since log J ≥ log 2 ≥ 0.5 , we have

P ( | e ^ k − e k | > ε ) ≤ P ( | I 1 | > ε 3 ) + P ( | I 2 | > ε 3 ) + P ( | I 3 | > ε 3 )

For I 1 , we have

P ( | I 1 | > ε 3 ) ≤ ∑ r = 1 R     P ( | ( p r − p ^ r ) ( p r + p ^ r ) | > ε 3 ) ≤ ∑ r = 1 R     P ( | p r − p ^ r | > c 1 ε 3 R J 3 ) ≤ R J 3 2 exp { − 6 n ( c 1 ε 3 R J 3 ) 2 3 + 4 ( c 1 ε 3 R J 3 ) }

For I 2 , we have

P ( | I 2 | > ε 3 ) ≤ ∑ j J k     P ( | w ^ k , j − w k , j | > c 1 ε 3 J 3 ) ≤ J 3 2 exp { − 6 n ( c 1 ε 3 J 3 ) 2 3 + 4 ( c 1 ε 3 J 3 ) }

For I 3 , we have

I 3 = ∑ j = 1 J k     ∑ r = 1 R [ ( w ^ k , j p ^ k , j r + w k , j p k , j r ) ( p ^ k , j r − p k , j r ) + p ^ k , j r p k , j r ( w k , j − w ^ k , j ) ] = ∑ j J k     ∑ r = 1 R [ ( w ^ k , j p ^ k , j r + w k , j p k , j r ) ( p ^ k , j r − p k , j r ) ] + ∑ j J k     ∑ r = 1 R     p ^ k , j r p k , j r ( w ^ k , j − w k , j ) : = I 31 + I 32

For I 31 and I 32 , we have

P ( | I 3 | > ε 3 ) ≤ P ( | I 31 | > ε 6 ) + P ( | I 32 | > ε 6 )

P ( | I 31 | > ε 6 ) ≤ ∑ j J k     ∑ r = 1 R     P ( | ( w ^ k , j p ^ k , j r + w k , j p k , j r ) ( p ^ k , j r − p k , j r ) | > ε 6 ) ≤ ∑ j J k     ∑ r = 1 R     p ( | p ^ k , j r − p k , j r | > c 1 ε 6 R J 3 ) ≤ R J 3 2 exp { − 6 n ( c 1 ε 6 R J 3 ) 2 3 + 4 ( c 1 ε 6 R J 3 ) }

P ( | I 32 | > ε 6 ) ≤ ∑ j J k     ∑ r = 1 R     P ( | p ^ k , j r p k , j r ( w ^ k , j − w k , j ) | > ε 6 ) ≤ ∑ j J k     ∑ r = 1 R     P ( | w ^ k , j − w k , j | c 1 ε 6 R J 3 ) ≤ R J 3 2 exp { − 6 n ( c 1 ε 6 R J 3 ) 2 3 + 4 ( c 1 ε 6 R J 3 ) }

In a word, we have the inequality,

P ( | e ^ k − e k | > 2 ε ) ≤ O ( R J 3 ) exp { − c 5 n ε 2 R 2 J 6 }

where c 5 represents a positive constant.

Proof of Theorem 3.1. By Conditions (C1) to (C3) and Lemma 3, we can get

P ( D ⊆ D ^ ) ≥ P ( | e ^ k − e k | ≤ c n − τ , ∀ k ∈ D ) ≥ P ( max 1 ≤ k ≤ P | e ^ k − e k | ≤ c n − τ ) ≥ 1 − ∑ k = 1 P     P ( max 1 ≤ k ≤ P | e ^ k − e k | > c n − τ ) ≥ 1 − O ( R J 3 ) p exp { − c 5 c 2 n 1 − 2 τ R 2 J 6 } ≥ 1 − O ( p exp { − b n 1 − 2 τ − 2 ε − 2 κ + ( ε + κ ) log n } )

Where b is a positive constant.

Lemma 4 (Lemma A.2 [19]). For any continuous covariate X k satisfying conditions (C4) and (C5), let F k ( y , x ) be the cumulative distribution function of ( Y , X k ) and F ^ k ( y , x ) be the empirical cumulative distribution function, we have P ( | F ^ k ( r , q ^ k , ( j ) ) − F k ( r , q k , ( j ) ) | > ε ) ≤ c 6 exp { − c 7 n 1 − 2 ρ ε 2 } for any ε > 0 , 1 ≤ r ≤ R and 1 ≤ j ≤ J k , where c 6 and c 7 are two positive constants.

Lemma 5 (Lemma A.3 [19]). Under (C1), (C4) and (C5), for any 0 < ε < 1 , so for continuous X k , we have P ( | e ^ k − e k | > 2 ε ) ≤ O ( R J ) exp { − c 9 n 1 − 2 ρ ε 2 R 4 J 4 } , there exists a positive constant c 9 .

Proof of Theorem 3.2. According to Lemma 4 and Lemma 5, the proof of Theorem 3.2 is the same as Theorem 3.1 and hence is omitted.

Proof of Theorem 3.3. According to Lemma 3 and 5 and under Conditions (C1), (C4), (C5) and (C7). The proof of Theorem 3.2 is proved by [19] and hence is omitted.