With the advancement of data acquisition tools and the improvement of computer storage capacity, ultra-high dimensional data has been widely applied in various scientific research fields, especially in genomics, tumor classification, machine learning and other fields. The traditional data screening methods can no longer be applied, and the existing feature screening methods for ultra-high dimensional data have their own limitations. Among them, when the covariates are both categorical and continuous variables, there are fewer research methods and the screening effect needs to be improved, so there is an urgent need to develop new theoretical and statistical methods to deal with ultra-high dimensional data. The pioneering work by Fan and Lv (2008) [1] introduced the concept of sure independence screening (SIS) in their seminal paper. Specifically, for linear regressions, they demonstrated that the approach based on Pearson correlation learning exhibits a sure screening property. This means that even when the

number of predictors (p) grows at a much faster rate than the number of observations (n) with logarithm of p equal to O ( n α ) for some α ∈ ( 0 , 1 2 ) , all relevant predictors can be selected with a probability approaching one (2009) [2] .

Numerous approaches have been developed in recent years for feature screening in ultrahigh-dimensional data. Wang (2009) [3] introduced forward regression as a method for handling such data. Fan and Song (2010) [4] applied maximum marginal likelihood estimates or maximum marginal likelihood to ultrahigh-dimensional screening in generalized linear models. Fan et al. (2011) [5] extended correlation learning to marginal nonparametric learning. Li et al. (2012) [6] presented a robust rank correlation screening method based on the Kendall correlation coefficient. He et al. (2013) [7] developed a quantile-adaptive framework for nonlinear variable screening in high-dimensional heterogeneous data. Fan et al. (2014) [8] introduced nonparametric independence screening, which selects variables based on the nonparametric marginal contributions of each covariate given the exposure variable. Nandy et al. (2022) [9] introduced covariate information number sure independence screening, which incorporates a marginal utility connected to the traditional Fisher information. Tong et al. (2022) [10] propose a model-free conditional feature screening method for ultra-high-dimensional data based on false discovery rate (FDR) control, which does not require a specific functional form of the regression function and is robust to heavy-tail responses and predictors.

To tackle the challenge of ultrahigh-dimensional feature screening in classification problems, Fan and Fan (2008) [11] introduced the t-test statistic for the two-sample mean problem as a marginal utility for feature screening and established its theoretical properties. Mai and Zou (2013) [12] applied the Kolmogorov filter to ultrahigh-dimensional binary classification. Cui et al. (2015) [13] proposed a screening procedure that utilizes empirical conditional distribution functions. Lai et al. (2017) [14] developed a feature screening procedure based on the expected conditional Kolmogorov filter for binary classification problems.

However, the aforementioned screening methods assume that the data types are continuous. For categorical covariates, Huang et al. (2014) [15] devised a model-free discrete feature screening method based on Pearson Chi-square statistics and demonstrated its sure screening property, as mentioned in Fan et al. (2009) [2] . When all the covariates are binary, Ni and Fang (2016) [16] proposed a model-free feature screening procedure based on information entropy theory for multi-class classification. Ni et al. (2017) [17] further extended this by introducing a feature screening procedure based on weighted Adjusted Pearson Chi-square for multi-class classification. Sheng and Wang (2020) [18] introduced a novel model-free feature screening method based on the classification accuracy of marginal classifiers for ultrahigh-dimensional classification. Anzarmou et al. (2022) [19] presented a new model-free interaction screening method called Kendall Interaction Filter (KIF) for classification in high-dimensional settings.

Based on the aforementioned research on classification models, this paper introduces a model-free feature screening approach for ultrahigh-dimensional multi-classification that accommodates both categorical and continuous covariates. The proposed method utilizes the maximal information coefficient (MIC) to evaluate the predictive power of the covariates. For screening categorical covariates, we employ the maximal information coefficient (MIC) index, which is equivalent to information gain [16] . The feature screening procedure proposed in this paper is based on maximal information coefficient, specifically referred to as Maximal Information Coefficient Sure Independence Screening (MIC-SIS). The maximum mutual information coefficient can be directly used to categorize the data without any cut-off processing, and it overcomes the disadvantages of information gain, such as the difficulty of calculating the joint probability, not belonging to the measurement method, no way to normalize it, and not being able to compare the results of different data and it, and the screening results are more robust and have lower computational complexity. It first finds an optimal discretization method, and then converts the mutual information value into a measurement method, and the value range is between [0, 1]. MIC has the advantages of wide application range, low computational complexity and strong robustness. The MIC-SIS method is rigorously proven to possess the sure screening property, as originally proposed by Fan and Lv [1] , ensuring that all significant features can be identified. Through simulation results, the MIC-SIS approach demonstrates the satisfaction of the sure screening property when compared to existing feature screening methods.

The paper is organized as follows: Section 2 provides a detailed description of the proposed MIC-SIS method. Section 3 establishes the sure screening property of the method. In Section 4, numerical simulations and a real data analysis example are presented to assess the sure screening property of our approach. Concluding remarks are provided in Section 5, and all proofs are included in the Appendix.

Firstly, we introduce the concept of the maximal information coefficient (MIC), and subsequently, we propose a screening procedure that is based on the maximal information coefficient.

In the subsequent sections, we will establish the theoretical properties of the proposed independence screening procedure. Previous studies by Fan and Lv [1] , Ji and Jin [22] , Zhou and Wang [20] and Ni and Fang [16] have demonstrated that the sure screening property ensures the effectiveness of the independence screening procedure. Hence, it is crucial to establish the sure screening property for MIC-SIS. The following conditions are assumed to guarantee the sure screening property of the MIC-SIS procedure. Although they may not be the weakest conditions, they are primarily imposed to facilitate the technical proofs.

(C1) Let X = ( x 1 , x 2 , ⋯ , x p ) , where x i is drawn from an unknown distribution F i . Each distribution F i has an unknown Lebesgue probability density function pdf f i , with i = 1 , 2 , ⋯ , p . The conditions specified in Lemma 3 in the Appendix apply to these distributions.

(C2) There exists a positive constant 0 < κ < 2 , such that

sup 1 ≤ k ≤ p ∑ i = 1 n ∑ r = 1 R log p ( X i k , Y r ) p ( X i k ) p ( Y r ) = O ( n κ ) ,   a . e .

(C3) There exists a positive constant c > 0 and τ ; the minimum MIC of the active predictors satisfies min k ∈ D ω k ∗ ≥ 2 c n − τ ;

(C4) Both X and Y exhibit sub exponential tail probabilities that hold uniformly in p. Specifically, there exists a positive constant μ 0 such that, for all 0 < μ ≤ μ 0 , the following condition holds:

sup p max 1 ≤ k ≤ p E { exp ( μ ‖ X k ‖ 1 2 ) } < ∞ ,   E { exp ( μ ‖ Y ‖ q 2 ) } < ∞

(C5) There exist two positive constants c 1 and c 2 such that, c 1 / R ≤ p ( Y = r ) ≤ c 2 / R , c 1 + c 2 ≤ R , c 1 / R ≤ p ( X k = j , Y = r ) ≤ c 2 / R and c 1 / J k ≤ p ( X k = j ) ≤ c 2 / J k for every 1 ≤ J k ≤ J , 1 ≤ r ≤ R , and 1 ≤ k ≤ p .

(C6) There exist a positive constant c 3 , such that 0 < f k ( x | Y = r ) < c 3 for any 1 ≤ r ≤ R , and x in the domain of X k , where f k ( x | Y = r ) is the Lebesgue density function of X k conditional on Y = r .

(C7) There exist a positive constant c 4 and 0 ≤ ρ < 1 / 2 such that f k ( x ) ≥ c 4 n − ρ for any 1 ≤ k ≤ p and x in the domain of X k , where f k ( x ) is the Lebesgue density function of X k . Furthermore, f k ( x ) is continuous in the domain of X k .

(C8) lim inf n → ∞ { min k ∈ D ω ^ k ∗ − max k ∈ I ω ^ k ∗ } > δ , where δ > 0 is a constant.

According to Ji and Jin [22] and conditions in the Appendix, conditions (C1) and (C2) ensure that the estimated probabilities converge strongly and uniformly to the true probabilities. According to Fan and Lv [1] and Cui [13] , condition (C3) allows the minimum true signal to disappear to zero in the order of n − τ as the sample size goes to infinity. According to the sure screening property proposed by Zhou and Wang [20] , then condition (C4) is established. Condition (C5) guarantees that the proportion of each class of variables cannot be either extremely small or extremely large. A similar assumption is also made in condition (C5) in Huang [15] and Cui [13] . To ensure that the sample percentiles are close to the true percentiles, condition (C6) excludes the extreme case that some X k put heavy mass in a small range. Condition (C7) requires the n − ρ as a lower bound on the density. According to Cui [13] , it is easy to show that ω k ∗ > 0 for k ∈ D and ω k ∗ = 0 for k ∈ I naturally holds. Thus, condition (C8) is established, and MIC index is able to separate active and inactive predictors well at the population level.

Theorem 1 (Sure Screening Property).

Under conditions(C1) - (C4), there exists the positive constant C 1 such that

P ( ω ^ k ∗ − ω k ∗ ≥ c n − τ ) ≤ O ( p ) exp { − C 1 n 1 − 2 τ }

Further, we have that

P ( D ⊆ D ^ ∗ ) ≥ 1 − O ( s n exp ( − C 1 n 1 − 2 τ ) ) .

In the equation above, s n represents the cardinality of D. According to Theorem 1, it implies that we can handle the ultra-high-dimensional scenario where the logarithm of p is on the order of O ( n 1 − 2 τ ) , with τ > 0 .

Theorem 2 (Ranking consistency property).

Under conditions(C5) - (C8), if log R J log n = O ( 1 ) and max { log P , log n } R 4 J 4 n 1 − 2 ρ = o ( 1 ) , then

lim inf n → ∞ { min k ∈ D ω k ∗ − max k ∈ I ω k ∗ } > 0 ,   a . s .

Theorem 2 demonstrates that the proposed screening index effectively distinguishes between active and inactive covariates at the sample level.

In practical scenarios, it is common to encounter datasets with a combination of continuous and categorical covariates, along with categorical responses. However, the available screening methods for such cases are limited. To address this problem, we introduce a new method called MIC-SIS (Maximum Information Coefficient-based Screening), which does not require continuous variables to be sliced and can be applied directly to a wide range of variables, overcoming the shortcomings of existing methods. In this paper, we demonstrate that MIC-SIS has theoretical properties such as deterministic screening and ranking consistency, and that no modeling is required. Through numerical simulations, we find that MIC-SIS can effectively screen covariates with better screening and lower computational complexity than existing methods. It also performs well in the empirical analysis of GLIOMA data.

One of the current challenges in covariate screening is missing data. It is common to have missing or incorrect data during the data collection process, which can affect the variable screening results. In future work, we aim to develop a new approach to feature screening that can either deal with missing variables prior to screening, or use classification models to screen features based on response variables.

The work was supported by National Natural Science Foundation of China [grant number 71963008].

The GLIOMA biological data that support the findings of this study are available from the feature selection database of Arizona State University (http://featureselection.asu.edu/).

The authors declare no conflicts of interest regarding the publication of this paper.

Chen, T.T. and Deng, G.M. (2023) Model-Free Feature Screening via Maximal Information Coefficient (MIC) for Ultrahigh-Dimensional Multiclass Classification. Open Journal of Statistics, 13, 917-940. https://doi.org/10.4236/ojs.2023.136046

Proof of Theoretical Result.

To establish the validity of Theorem 1, we rely on three accompanying lemmas. The first two lemmas furnish us with exponential inequalities, and their detailed proofs can be found in [25] .

Lemma 1. Let μ = E ( Y ) . If P ( a ≤ Y ≤ b ) = 1 , then

E [ exp { s ( Y − μ ) } ] ≤ exp { s 2 ( b − a ) 2 8 } .

Lemma 2. Let h ( Y 1 , ⋯ , Y m ) be a kernel function for the U statistics U n , and θ = E { h ( Y 1 , ⋯ , Y m ) } . If a ≤ h ( Y 1 , ⋯ , Y m ) ≤ b holds, then for any t > 0 and n ≥ m , we have the following inequality

P ( U n − θ ≥ t ) ≤ exp ( − 2 [ n m ] t 2 ( b − a ) 2 ) .

where [ n / m ] denotes the integer part of n / m .

Lemma 2 represents the one-sided tail inequality of U n . As a result of its symmetry, we can readily derive the two-sided tail inequality of U n

P ( | U n − θ | ≥ t ) ≤ 2 exp ( − 2 [ n m ] t 2 ( b − a ) 2 ) .

Lemma 3. (The asymptotic property of nonparametric density estimators). Suppose that f ″ ( x ) exists and h = c n − 1 5 , then

n 2 5 { p ^ ( x ) − p ( x ) } → L N ( c 2 2 f ″ ( x ) μ 2 ( K ) , 1 c f ( x ) ‖ K ‖ 2 2 ) .

From the above equation, μ 2 ( K ) = ∫ s 2 K ( s ) d s and ‖ K ‖ 2 2 = ∫ K 2 ( s ) d s .

Lemma 3 directly implies that p ^ ( x ) → p p ( x ) . Under certain additional stringent conditions, we can achieve strong uniform convergence of p ^ ( x ) .

lim n → ∞ sup x | p ^ ( x ) − p ( x ) | = 0 ,   a . e .

For more detailed information regarding the strong uniform convergence, one can refer to references such as [26] or [27] . These sources provide further insights and explanations on the topic.

Proof of Theorem 1. We begin by demonstrating that the following inequality holds for each k:

P { | ω ^ k ∗ − ω k ∗ | ≥ c n − τ } ≤ O ( exp ( − C 1 n 1 − 2 τ ) ) .

Since ω ^ k ∗ is obtained by normalizing ω ^ k without altering its properties, we only need to establish that the aforementioned inequality holds for each k:

P { max a ∗ b < B | ω ^ k − ω k | log 2 min ( a , b ) ≥ c n − τ } ≤ O ( exp ( − C 1 n 1 − 2 τ ) ) .

Because

max a ∗ b < B log 2 min ( a , b ) | ω ^ k − ω k | = | ω ^ k − ω k | = | ∑ i = 1 n ∑ r = 1 R p ^ ( X i k , Y r ) log p ^ ( X i k , Y r ) p ^ ( X i k ) p ^ ( Y r ) − ∫ p ( x k , y ) log p ( x k , y ) p ( x k ) p ( y ) d x k d y | = | ∑ i = 1 n ∑ r = 1 R p ^ ( X i k , Y r ) log p ^ ( X i k , Y r ) p ^ ( X i k ) p ^ ( Y r ) − 1 n 2 ∑ i = 1 n ∑ r = 1 R log p ^ ( X i k , Y r ) p ^ ( X i k ) p ^ ( Y r )         + 1 n 2 ∑ i = 1 n ∑ r = 1 R log p ^ ( X i k , Y r ) p ^ ( X i k ) p ^ ( Y r ) − p ( x k , y ) log ∫ p ( x k , y ) p ( x k ) p ( y ) d x k d y | = | M k , 1 + M k , 2 | .

Using Lemma 3 and the strong law of large numbers, we observe the convergence of

M k , 1 = ∑ i = 1 n ∑ r = 1 R p ^ ( X i k , Y r ) log p ^ ( X i k , Y r ) p ^ ( X i k ) p ^ ( Y r ) − 1 n 2 ∑ i = 1 n ∑ r = 1 R log p ^ ( X i k , Y r ) p ^ ( X i k ) p ^ ( Y r ) → 0 ,   a . e .

Next, our goal is to establish an upper bound for the second term.

Define h ( X i k , Y r ; X k , Y ) = log p ^ ( X i k , Y r ) p ^ ( X i k ) p ^ ( Y r ) as the kernel of the U statistics of I k , 2 ∗ , where we define M k , 2 = I k , 2 − ω k , and

I k , 2 ∗ = I k , 2 − 1 n 2 ∑ i = 1 n ∑ r = 1 R log p ^ ( X i k , Y r ) p ^ ( X i k ) p ^ ( Y r ) .

By applying Markov's inequality, we can ensure that:

P ( I k , 2 ⋆ − E h > ε ) ≤ exp ( − t ε ) exp ( − t E h ) E [ exp ( t I k , 2 ⋆ ) ] ,   for   any   t > 0 .

where E h = ∫ p ( x k , y ) log p ( x k , y ) p ( x k ) p ( y ) d x k d y .

Following the approach utilized by Li et al. (2012) [28] to handle the U statistics, and considering condition (C2), we can immediately deduce that:

P ( I k , 2 ∗ − E h > ε ) ≤ exp ( − t ε + t 2 8 n ) .

By selecting t = 4 n ε , we obtain P ( I k , 2 ∗ − E h > ε ) ≤ exp ( − 2 n ε 2 ) . Consequently, taking into account the symmetry of U statistics, we can derive the bilateral tail inequality:

P ( | I k , 2 ∗ − E h | > ε ) ≤ 2 exp ( − 2 n ε 2 ) .

By utilizing the relationship between I k , 2 ∗ and I k , 2 , we can demonstrate that

P ( | M k , 2 | > 2 ε ) = P ( | I k , 2 − ω k ∗ | > 2 ε ) = P ( | I k , 2 ∗ + 1 n 2 ∑ i = r log p ^ ( X i k , Y r ) p ^ ( X i k ) p ^ ( Y r ) − ω k ∗ | > 2 ε ) .

Under condition (C2), for ε > 0 , we can choose a sufficiently large N 1 such that when n > N 1 , 1 n 2 ∑ i = r log p ^ ( X i k , Y r ) p ^ ( X i k ) p ^ ( Y r ) < ε 3 . Furthermore, we can easily establish that

P ( | I k , 2 − ω k ∗ | > 2 ε ) ≤ P ( | I k , 2 ⋆ − ω k ∗ | > 5 3 ε ) .

Note that

| I k , 2 − ω k ∗ | = | I k , 2 − E h + E h − ω k ∗ | .

Similarly, employing the same technique and selecting a larger N 2 , we can ensure that when n > N 2 , | ω k ∗ − E h | < ε 3 . This directly implies that

P ( | I k , 2 ∗ − ω k ∗ | > 5 3 ε ) ≤ P ( | I k , 2 ∗ − E h | > 4 3 ε ) .

Let ε = c n − τ , 0 < τ < 1 2 . By employing P ( | I k , 2 ∗ − E h | > ε ) ≤ 2 exp ( − 2 n ε 2 ) , together with Lemma 1 and Bonferroni’s inequality, we can deduce that

P { | ω ^ k ∗ − ω k ∗ | ≥ c n − τ } ≤ 2 exp ( − 2 c 2 n 1 − 2 τ ) .

We thus have

P { max 1 ≤ k ≤ p | ω ^ k ∗ − ω k ∗ | ≥ c n − τ } ≤ 2 p exp ( − 2 c 2 n 1 − 2 τ ) = O ( p [ exp ( − C 1 n 1 − 2 τ ) ] ) .

Next, we prove the second part of Theorem 1.

If D ⊆ D ^ ∗ , it implies that there must exist some k ∈ D such that ω ^ k ∗ = c n − τ . By utilizing condition (C3), we can deduce that if ω ^ k ∗ = c n − τ holds for some k ∈ D , then ω ^ k ∗ = c n − τ also holds for some k ∈ D . Thus, the event { D ⊆ D ^ ∗ } is a subset of the event { | ω ^ k ∗ − ω k ∗ | ≥ c n − τ ,   for   som e   k ∈ D } . Taking the complement on both sides, we obtain { max k ∈ D | ω ^ k ∗ − ω k ∗ | ≤ c n − τ } is a subset of { D ⊆ D ^ ∗ } . Therefore, we have:

P ( D ⊆ D ^ ∗ ) ≥ P { max k ∈ D | ω ^ k ∗ − ω k ∗ | ≤ c n − τ } = 1 − P { min k ∈ D | ω ^ k ∗ − ω k ∗ | ≥ c n − τ } = 1 − s n P { | ω ^ k ∗ − ω k ∗ | ≥ c n − τ } ≥ 1 − O ( s n [ exp ( − C 1 n 1 − 2 τ ) ] ) .

In the above equation, s n is the cardinality of D.

Now we prove Theorem 2. The proofs for Lemma 4 and Lemma 5 can be found in Ni and Fang (2016) [16] .

Lemma 4. For categorical covariates X k and response Y,under condition (C5), for any 0 < ε < 1 , we have P ( | ω ^ k ∗ − ω k ∗ | > 2 ε ) ≤ O ( R J ~ ) exp { − c 5 n ε 2 R 4 J 4 } , where c 5 represents a positive constant.

Lemma 5. For continuous covariates X k and response Y, under condition (C5), (C6) and (C7), for any 0 < ε < 1 , we have P ( | ω ^ k ∗ − ω k ∗ | > 2 ε ) ≤ O ( R J ~ ) exp { − c 6 n 1 − 2 ρ ε 2 R 4 J 4 } , where c 6 represents a positive constant.

Under Conditions (C5)-(C8) and by Lemmas 4 and 5, if log R J log n = O ( 1 ) and max { log P , log n } R 4 J 4 n 1 − 2 ρ = o ( 1 ) , we get

P ( min k ∈ D ω ^ k ∗ − max k ∈ I ω ^ k ∗ < δ 2 ) ≤ P ( ( min k ∈ D ω ^ k ∗ − max k ∈ I ω ^ k ∗ ) − ( min k ∈ D ω k ∗ − max k ∈ I ω k ∗ ) < − δ 2 ) ≤ P ( | ( min k ∈ D ω ^ k ∗ − max k ∈ I ω ^ k ∗ ) − ( min k ∈ D ω k ∗ − max k ∈ I ω k ∗ ) | > δ 2 ) ≤ P ( max 1 ≤ k ≤ p | ω ^ k ∗ − ω k ∗ | > δ 4 ) ≤ O ( R J k ) p exp { − c 5 n 1 − 2 ρ R 4 J k 4 } = O ( exp { log R J + log p − c 7 n 1 − 2 ρ R 4 J 4 } )

where c 7 = min { c 5 , c 6 } ( δ / 4 ) 2 . Since log R J log n = O ( 1 ) , there exists a positive constant c 8 such that log ( R J ) ≤ c 8 log n . Also, max { log P , log n } R 4 J 4 n 1 − 2 ρ = o ( 1 ) implies that log p ≤ 1 2 c 7 n 1 − 2 ρ R 4 J 4 and 1 2 c 7 n 1 − 2 ρ R 4 J 4 ≥ ( c 8 + 2 ) log n for large n. Then there exists a n 0 such that ∑ n = n 0 ∞ exp { log R J + log p − c 7 n 1 − 2 ρ R 4 J 4 } ≤ ∑ n = n 0 ∞ exp { c 8 log n − 1 2 c 7 n 1 − 2 ρ R 4 J 4 } ≤ ∑ n = n 0 ∞ exp { c 8 log n − ( c 8 + 2 ) log n } = ∑ n = n 0 ∞ n − 2 < ∞ . According to Ni and Fang (2016) [16] and by the Borel Cantelli Lemma, we can get lim inf n → ∞ { min k ∈ D ω k ∗ − max k ∈ I ω k ∗ } ≥ δ 2 > 0 , a.s.

This is the end of the proof.