Assuming the complete random vector Χ ∈ ℝ p has the covariance matrix Σ X = ( σ ^ i j ) p × p . Using a p -dimensional random vector F to represent noisy data, the noisy model can be expressed as

F = X + ε , (3)

where ε ∈ ℝ p represents noise. In this section, we hope to build a p × p matrix Σ ^ F based on n independent random noisy samples F 1 , F 2 , ⋯ F n ∈ ℝ p of F . We next use Σ ^ F to estimate the covariance matrix Σ X of the random vector X .

The noisy sample with missing data are represented by F * = { F 1 * , F 2 * , ⋯ , F n * } , where F i * = ( F 1 i S 1 i , F 2 i S 2 i , ⋯ , F p i S p i ) T is the i th observation sample. The definition of the generalized sample mean is as follows:

F ¯ * : = ( F ¯ i * ) 1 ≤ i ≤ p , F ¯ i * = 1 n i * ∑ k = 1 n F i k * = 1 n i * ∑ k = 1 n F i k S i k ,

the generalized sample covariance matrix is defined as

Σ ^ F ∗ : = ( σ ^ i j * ( F ) ) 1 ≤ i , j ≤ p , σ ^ i j * ( F ) = 1 n i j * ∑ k = 1 n ( F i k − F ¯ i * ) ( F j k − F ¯ j * ) S i k S j k . (4)

Two new assumptions in [12] are presented below.

Assumption 1. The observation index S : = { S 1 , S 2 , ⋯ S n } can be random or deterministic, but it is independent of the noisy observation sample F : = { F 1 , F 2 , ⋯ F n } .

Assumption 2. The random vectors F 1 , F 2 , ⋯ F n are i.i.d., where F k = X k + ε k , and

X k = Γ Z k + μ , ε k = Γ ε Z k , k = 1 , 2 , ⋯ n .

μ represents a fixed p -dimensional mean vector. Γ , Γ ε ∈ ℝ p × q ( p ≤ q ) are fixed matrices with Γ Γ T = Σ and Γ ε Γ ε T = Σ ε . Each component of the random vector Z k = ( Z 1 k , Z 2 k , ⋯ , Z q k ) T i.i.d. sub-Gaussian with a variance of 1 and a mean of 0. For any s > 0 , there exists a parameter τ > 0 such that E ( e s Z i k ) ≤ e τ s 2 / 2 , that is, Z i k ∼ S u b ( τ ) .

The hard thresholding estimator based on complete data was proposed by [1] . When most of the elements in each row or column of the true covariance matrix are close to zero or negligible, set the elements of the sample covariance matrix below a certain threshold to 0, and leave the remaining elements unaltered to estimate the true covariance matrix, so as to reduce the error. In [2] ,

P { | σ ^ i j − σ i j | ≤ t } ≥ 1 − C p − 8

for t = γ log p / n , where C is a constant. The threshold is set to γ log p / n .

In this paper, it is extended to the case of missing and noisy data. According to Lemma 4.6 in [12] , if Assumption 1 and Assumption 2 are both hold, then there are two absolute constants C and c greater than 0, such that

P { | σ ^ i j * ( F ) − σ i j | ≤ x } ≥ 1 − C exp { − c n min ∗ min ( x 2 τ 4 σ i i σ j j , x τ 2 σ i i σ j j ) } (5)

for any x > 0 . Since σ i i σ j j ≤ ρ 2 , the above can be simplified to: there are constants C > 0 and γ > 0 , such that

P { | σ ^ i j * ( F ) − σ i j | ≤ x } ≥ 1 − C exp ( − 8 γ 2 n min * x 2 ) . (6)

Where x ≤ ρ , and the constants C and γ only depend on ρ . Note that Inequality can be written as P { | σ ^ i j * ( F ) − σ i j | ≤ x } ≥ 1 − C p − 8 when x = γ log p / n min * .

The hard thresholding estimator Σ ^ F t h of the covariance matrices Σ X ∈ G q ( ρ , c n , p ) is defined by transforming the generalized sample covariance σ ^ i j * ( F ) in Equation (4),

Σ ^ F t h = ( σ ^ i j t h ( F ) ) p × p = ( σ ^ i j * ( F ) ⋅ I ( | σ ^ i j * ( F ) | ≥ λ ) ) p × p , λ = γ log p n min * , (7)

where γ is a constant and γ > 0 .

The following is Lemma 1, which plays an important role in studying the minimax upper bound. Lemma 1 generalizes Lemma 8 in [8] from complete to noisy and missing sample.

Lemma 1. Define event A i j : = { | σ ^ i j t h ( F ) − σ i j | > 4 min ( | σ i j | , λ = γ log p / n min * ) } , then there is constant c > 0 , which only depends on ρ , such that

P { A i j } ≤ 2 c p − 9 2 .

Proof : Firstly, define event B 1 : = { | σ ^ i j t h ( F ) | ≥ λ } . It is easy to know that

B 1 ⊂ { | σ ^ i j t h ( F ) − σ i j | ≥ | σ ^ i j t h ( F ) | − | σ i j | ≥ λ − | σ i j | } , B 1 c ⊂ { | σ ^ i j t h ( F ) − σ i j | ≥ | σ i j | − | σ ^ i j t h ( F ) | > | σ i j | − λ } . (8)

According to the definition of σ ^ i j t h ( F ) in Equation (7),

| σ ^ i j t h ( F ) − σ i j | = | σ ^ i j ∗ ( F ) − σ i j | I ( B 1 ) + | σ i j | I ( B 1 c ) . (9)

Next, we will prove this lemma in different cases. It can be obtained by simple calculation:

P { | σ ^ i j t h ( F ) − σ i j | ≤ 4 min ( | σ i j | , λ ) } { ≥ 1 − C p − 9 / 2 , | σ i j | < λ / 4 , ≥ 1 − C p − 8 , λ / 4 ≤ | σ i j | ≤ 2 λ , ≥ 1 − 2 C p − 8 | σ i j | > 2 λ .

Therefore, there exists a constant c > 0 , such that

P { | σ ^ i j t h ( F ) − σ i j | > 4 min ( | σ i j | , λ ) } ≤ 2 c p − 9 / 2 .

Next, we can obtain the upper bound for estimating the sparse covariance matrices by utilizing the risk properties of thresholding estimator.

Theorem 1. If Assumption 1 and Assumption 2 hold, log p = o ( n min * ) and p ≥ n min * , then there is a constant C > 0 such that the hard thresholding estimator Σ ^ F t h defined by Equation (7) satisfies

sup Σ X ∈ G q ( ρ , c n , p ) E ‖ Σ ^ F t h − Σ X ‖ 1 2 ≤ C c n , p 2 ( log p n min * ) 1 − q . (10)

Proof: Easy to know, ‖ Σ ^ F t h − Σ X ‖ 1 2 = [ max j ∑ i = 1 p | σ ^ i j t h ( F ) − σ i j | ] 2 . If event A i j occurs,

∑ i = 1 p | σ ^ i j t h ( F ) − σ i j | > ∑ i = 1 p 4 min ( | σ i j | , λ ) .

Simple calculations show that

∑ i = 1 p min ( | σ i j | , λ ) = ( ∑ i ≤ k ′ + ∑ i > k ′ ) min ( | σ i j | , λ ) ≤ ( ∑ i ≤ k ′ + ∑ i > k ′ ) min ( | σ [ i ] j | , γ log p n min * ) .

According to the definition of G q ( ρ , c n , p ) , we know that max 1 ≤ j ≤ p { | σ [ i ] j | q } ≤ c n , p / i , so | σ [ i ] j | ≤ ( c n , p / i ) 1 / q . Select the constant k ′ to satisfy k ′ = ⌊ c n , p ( n min * / log p ) q / 2 ⌋ , so

∑ i = 1 p min ( | σ i j | , λ ) ≤ ∑ i ≤ k ′ min ( ( c n , p i ) 1 q , γ log p n min * ) + ∑ i > k ′ min ( ( c n , p i ) 1 q , γ log p n min * ) ≤ k ′ γ log p n min * + ∑ i > k ′ ( c n , p i ) 1 q ≤ C 1 c n , p ( log p n min * ) 1 − q 2 .

Let the matrix D = ( d i j ) p × p satisfy d i j = | σ ^ i j t h ( F ) − σ i j | I ( A i j ) , we have

E ‖ Σ ^ F t h − Σ X ‖ 1 2 ≤ 2 E ‖ Σ ^ F t h − Σ X − D ‖ 1 2 + 2 E ‖ D ‖ 1 2 .

Then, it is straightforward to acquire

2 E ‖ Σ ^ F t h − Σ X − D ‖ 1 2 = 2 E [ max j ∑ i = 1 p | σ ^ i j t h ( F ) − σ i j − ( σ ^ i j t h ( F ) − σ i j ) I ( A i j ) | ] 2 ≤ 2 E [ max j ∑ i = 1 p | ( σ ^ i j t h ( F ) − σ i j ) I ( A i j c ) | ] 2 ≤ C 2 c n , p 2 ( log p n min * ) 1 − q .

Therefore, we only need to prove that 2 E ‖ D ‖ 1 2 is negligible.

Firstly,

E ‖ D ‖ 1 2 = E [ max j ∑ i = 1 p | d i j | ] 2 ≤ p 2 E ( max i , j d i j ) 2 = p 2 E [ ( max i , j d i j 2 ) I ( A i j ∩ { | σ ^ i j * ( F ) | ≥ λ } ) ] + p 2 E [ ( max i , j d i j 2 ) I ( A i j ∩ { | σ ^ i j * ( F ) | < λ } ) ] ≤ p 2 E [ ( max i , j | σ ^ i j * ( F ) − σ i j | ) 2 I ( A i j ) ] + p 2 E [ ( max i , j | σ i j | ) 2 I ( A i j ) ] = E 1 + E 2 .

According to the Cauchy-Schwartz inequality, we know that | σ i j | 2 ≤ σ i i σ j j , and because P { A i j } ≤ 2 c p − 9 / 2 , p ≥ n min * , so

E 2 ≤ p 2 E ( max i , j σ i i σ j j ) P ( A i j ) ≤ p 2 ρ 2 2 c p − 9 2 ≤ C 3 p − 5 2 ≤ C 4 n min * .

In addition, E 1 ≤ p 2 max i , j E | σ ^ i j * ( F ) − σ i j | 2 P ( A i j ) . From Inequality (5),

E | σ ^ i j * ( F ) − σ i j | 2 = ∫ | σ ^ i j * ( F ) − σ i j | 2 d P ≤ ∫ 0 ∞ x P { | σ ^ i j * ( F ) − σ i j | ≥ x } d x ≤ ∫ 0 ρ x P { | σ ^ i j * ( F ) − σ i j | ≥ x } d x + ∫ ρ ∞ x P { | σ ^ i j * ( F ) − σ i j | ≥ x } d x ≤ ρ 2 + ∫ ρ ∞ x exp ( − C 5 n min ∗ x ρ ) d x ≤ ρ 2 + ρ 2 C 5 n min ∗ exp ( − C 5 n min ∗ ) .

From Lemma 1, we have P { A i j } ≤ 2 c p − 9 / 2 and p ≥ n min * , so

E 1 ≤ p 2 max i , j E | σ ^ i j * ( F ) − σ i j | 2 P ( A i j ) ≤ p 2 ( ρ 2 + ρ 2 C 5 n min ∗ exp ( − C 5 n min ∗ ) ) 2 c p − 9 / 2 ≤ p 2 ρ 2 2 c p − 9 / 2 + p 2 ρ 2 C 5 n min ∗ exp ( − C 5 n min ∗ ) 2 c p − 9 / 2 ≤ C 4 n min * + C 6 n min * ≤ C 7 n min *

To sum up,

E ‖ D ‖ 1 2 ≤ E 1 + E 2 ≤ C 8 n min * = O ( 1 n min * ) .

Before studying the lower bound, introduce some useful lemmas and symbols.

Lemma 2. Assume P and V are two probability measures, with p and v representing their probability density functions. The total variation distance between P and V is V ( P , V ) = 1 − ∫ min ( d P , d P ) . Define the total variation affinity as ‖ P ∧ V ‖ : = ∫ min ( d P , d P ) = ∫ p ( x ) ∧ v ( x ) d x . The Kullback divergence between P and V is expressed as K L ( P ‖ V ) = ∫ p ( x ) log [ p ( x ) / v ( x ) ] d x . Thus, ‖ P ∧ V ‖ and K L ( P ‖ V ) satisfy the following inequality:

1 − ‖ P ∧ V ‖ ≤ K L ( P ‖ V ) 2 . (11)

Lemma 2 in [14] and Le Cam’s lemma and its corollary in [2] [12] introduced below are important tools for proving minimax lower bound.

Lemma 3 (Le Cam). Suppose Θ = { θ 0 , θ 1 , ⋯ , θ m } is a finite set of parameters. Let L be a loss function and l min : = min 1 ≤ i ≤ m inf t [ L ( t , θ 0 ) + L ( t , θ i ) ] , then

sup θ ∈ Θ E [ L ( θ ˜ , θ ) ] ≥ 1 2 l min ‖ P θ 0 ∧ P ¯ ‖ .

θ ˜ is any estimator of θ based on the observed values of the probability measure P θ ( θ ∈ Θ ) , and P ¯ = 1 m ∑ i = 1 m P θ i .

Lemma 4. Suppose Σ ˜ be any estimator of Σ i based on the collection of probability measures { P Σ 0 , P Σ 1 , ⋯ , P Σ m } . We get

sup 1 ≤ i ≤ m E ‖ Σ ˜ − Σ i ‖ 1 ≥ 1 2 ‖ P Σ 0 ∧ P ¯ ‖ ⋅ inf 1 ≤ i ≤ m ‖ Σ i − Σ 0 ‖ 1 ,

where P ¯ = 1 m ∑ i = 1 m P Σ i .

Before studying the minimax risk lower bound, it is advisable to construct a matrix with all off-diagonal elements equal to 0 except the first row or column. Let H be the collection of p × p symmetric matrices in which exactly k non-diagonal elements in the first row or column equal to 1 and all other elements are 0. Let k = ⌊ c n , p ( n 0 / log p ) q / 2 ⌋ . Define

G 0 = { Σ = ( σ i j ) 1 ≤ i , j ≤ p : Σ = I p 或 Σ = I p + a H , H ∈ H } , (12)

where I p represents the identity matrix of size p × p , a = δ log p / n 0 , and δ is a constant. Assuming ρ > 1 , 0 < δ < min { 1 , 1 / ( 4 M ) } , it is easy to know that G 0 ⊂ G q ( ρ , c n , p ) .

Obtaining the lower bound requires two steps. Firstly, the subset of the parameter space constructed above is selected to simplify the proof. Secondly, calculate the total variation affinity between two probability measures.

Theorem 2. Let 1 ≤ n 0 ≤ n min * , p ≥ n 0 ν ( ν > 1 ) , and log p ≤ n 0 . Assume c n , p ≤ M ( n 0 / log p ) ( 1 − q ) / 2 with 0 ≤ q < 1 , M > 0 . For any 1 ≤ n 0 ≤ n min * , there exists a constant c > 0 such that the minimax risk lower bound for estimating the covariance matrix Σ X satisfies

inf Σ ˜ F sup Σ X ∈ G q ( ρ , c n , p ) E ‖ Σ ˜ F − Σ X ‖ 1 2 ≥ c c n , p 2 ( log p n 0 ) 1 − q . (13)

where Σ ˜ F is any estimator of Σ X based on noisy sample.

Proof: Assume G 0 = { Σ 0 , Σ 1 , ⋯ , Σ m ∗ } has m ∗ + 1 elements, where Σ 0 represents the identity matrix and Σ i , i = 1 , ⋯ , m ∗ represent the non-identity matrix, then m ∗ = C a r d ( G 0 ) − 1 = C p − 1 k .

Assume that X l , l = 1 , ⋯ , n ~ i . i . d . N ( 0 , Σ i ) , i = 1 , ⋯ , m ∗ , and the probability measure and probability density function are P Σ i and f i , respectively, that is, Σ X ∈ G 0 . Let F l ~ i . i . d . N ( 0 , Σ i ( F ) ) with Σ i ( F ) = Σ i + s 2 I p and P Σ i ( F ) is the probability measure. Since G 0 ⊂ G q ( ρ , c n , p ) , it is easy to know that

inf Σ ˜ F sup Σ X ∈ G q ( ρ , c n , p ) E ‖ Σ ˜ F − Σ X ‖ 1 2 ≥ inf Σ ˜ F sup Σ X ∈ G 0 E ‖ Σ ˜ F − Σ X ‖ 1 2 .

Therefore, to prove Inequality (13), just prove the following Inequality:

inf Σ ˜ F sup Σ X ∈ G 0 E ‖ Σ ˜ F − Σ X ‖ 1 2 ≥ c c n , p 2 ( log p n 0 ) 1 − q . (14)

Lemma 3.3 shows that

sup Σ X ∈ G 0 E ‖ Σ ˜ F − Σ ‖ 1 ≥ sup 1 ≤ i ≤ m ∗ E ‖ Σ ˜ F − Σ i ‖ 1 ≥ 1 2 ‖ P Σ 0 ( F ) ∧ P ¯ ( F ) ‖ ⋅ inf 1 ≤ i ≤ m ∗ ‖ Σ i ( F ) − Σ 0 ( F ) ‖ 1 .

Since a = δ log p / n 0 and k = ⌊ c n , p ( n 0 / log p ) q / 2 ⌋ , there exists a constant c 1 > 0 such that

inf 1 ≤ i ≤ m ∗ ‖ Σ i ( F ) − Σ 0 ( F ) ‖ 1 = inf 1 ≤ i ≤ m ∗ ‖ ( Σ i + s 2 I p ) − ( Σ 0 + s 2 I p ) ‖ 1 = inf 1 ≤ i ≤ m ∗ ‖ a H ‖ 1 = k a ≥ c n , p ( n 0 log p ) q / 2 ⋅ δ log p n 0 ≥ c 1 c n , p ( log p n 0 ) 1 − q 2 (15)

Obviously, to prove Inequality (14), we only need to prove that there is a constant c 2 > 0 such that ‖ P Σ 0 ( F ) ∧ P ¯ ( F ) ‖ ≥ c 2 .

From F l ~ i . i . d . N ( 0 , Σ i ( F ) ) , we have

K L ( P Σ 0 ( F ) ‖ P Σ i ( F ) ) = 1 2 [ t r ( Σ i − 1 ( F ) Σ 0 ( F ) ) − l o g det ( Σ i − 1 ( F ) Σ 0 ( F ) ) − p ] .

Let B = − a H , it is easy to know that Σ 0 ( F ) = Σ i ( F ) + B . Suppose the eigenvalues of B Σ i − 1 ( F ) are ξ 1 , ⋯ , ξ p , then there are

t r ( Σ i − 1 ( F ) Σ 0 ( F ) ) = t r [ Σ i − 1 ( F ) ( Σ i ( F ) + B ) ] = t r [ I p + B Σ i − 1 ( F ) ] = p + ∑ i = 1 p ξ i . (16)

In addition, we can know that

log det ( Σ i − 1 ( F ) Σ 0 ( F ) ) = ∑ i = 1 p log ( 1 + ξ i ) = t r ( Σ i − 1 ( F ) Σ 0 ( F ) ) − p − ∑ i = 1 p 1 2 ( 1 + θ ) ξ i 2 , (17)

where θ is a number between 0 and ξ i . Putting Equation (16) and Equation (17) into K L ( P Σ 0 ( F ) ‖ P Σ i ( F ) ) , we can get

K L ( P Σ 0 ( F ) ‖ P Σ i ( F ) ) = 1 2 [ p + ∑ i = 1 p 1 2 ( 1 + θ ) ξ i 2 − p ] = 1 2 ∑ i = 1 p 1 2 ( 1 + θ ) ξ i 2 ≤ 1 4 ∑ i = 1 p ξ i 2 .

According to Theorem 1.3 in [12] ,

∑ i = 1 p ξ i 2 = t r ( ( B Σ i − 1 ( F ) ) H B Σ i − 1 ( F ) ) = ‖ B Σ i − 1 ( F ) ‖ F 2 ≤ ‖ Σ i − 1 ( F ) ‖ s p 2 ‖ B ‖ F 2 ≤ 2 k a 2 .

It is easy to see that P ¯ ( F ) = 1 / m ∗ ∑ i = 1 m ∗ P Σ i ( F ) , hence

K L ( P Σ 0 ( F ) ‖ P ¯ ( F ) ) ≤ 1 m ∗ ∑ i = 1 m ∗ K L ( P Σ 0 ( F ) ‖ P Σ i ( F ) ) ≤ 1 8 .

Lemma 2 implies

‖ P Σ 0 ( F ) ∧ P ¯ ( F ) ‖ ≥ 1 − K L ( P Σ 0 ( F ) ‖ P ¯ ( F ) ) 2 = 3 4 .

That is, there exists a constant c 2 > 0 such that ‖ P Σ 0 ( F ) ∧ P ¯ ( F ) ‖ ≥ c 2 .

It is worth noting that Theorem 2 requires c n , p ≤ M ( n 0 / log p ) ( 1 − q ) / 2 ( M > 0 ) , which is a necessary condition. If c n , p > M ( n 0 / log p ) ( 1 − q ) / 2 , then

inf Σ ˜ F sup Σ X ∈ G q ( ρ , c n , p ) E ‖ Σ ˜ F − Σ X ‖ 1 2 ≥ inf Σ ˜ F sup Σ X ∈ G q ( ρ , M ( n 0 / log p ) ( 1 − q ) / 2 ) E ‖ Σ ˜ F − Σ X ‖ 1 2 ≳ M 2 .

Σ X does not have a consistent estimator in this case.

Theorem 1 and Theorem 2 show that the estimator Σ ^ F t h we construct is rate-optimal over G q ( ρ , c n , p ) under the l 1 norm.