We assume the MLR model using Equation (1):

O 3 i = β 0 + β 1 O 3 i − 1 + ∑ j = 2 p β j v a r m e t e o i j + e i (1)

where O 3 i : ozone concentration at day i; O 3 i − 1 : ozone concentration at day i − 1 (i.e. the persistence); v a r m e t e o i j : Meteorological variable j observed on day i.

Equation (1) can be written using usual matrix format after centring of the response variable:

y = X β + e (2)

where y is an ( n × 1 ) vector of centered dependant variable (O3 concentrations at day i), X is a ( n × p ) matrix of standardized predictors (Observed meteorological variables and O3 concentrations at day i − 1), β is an ( p × 1 ) vector of unknown regression coefficients and e is an ( n × 1 ) vector of random errors. Classically, the distribution of e is assumed to be normal with mean equal to 0 and a variance covariance matrix equal to σ 2 I ; where I is the identity matrix.

The usual unbiased Ordinary Least Squares (OLS) estimator is expressed by (3) [33] :

β ^ OLS = ( X T X ) − 1 X T y (3)

The prediction of y using OLS, y ^ OLS , is given by y ^ OLS = X β ^ OLS . It is well known that this estimator is likely to lead to an unstable model and poor predictions in presence of quasi-collinearity among the predictors or in the case of the small sample and high dimensional setting.

The principal components regression (PCR) approach involves running PCA on the predictor variables and, thereafter, using the first m principal components (PC) with 1 ≤ m ≤ p , F 1 , ⋯ , F m , as the predictors in a linear regression model [14] [34].

The appropriate number, m, of first principal components to be introduced in the model can be determined in practice by a validation technique such as Leave One Out cross validation (LOO). The PCR model can be written as in Equation (4):

y = F ( m ) β PCR ( m ) + e (4)

where F ( m ) is a matrix with m columns containing the m first PCs. The Ordinary Least Squares (OLS) estimator of β PCR is given by: β ^ PCR = ( F ( m ) T F ( m ) ) − 1 F ( m ) T y .

It is easy to express this model in terms of the original variables by remarking that F ( m ) = X U ( m ) , where U ( m ) ( U T U = I p ) is the matrix containing the m dominant normalised eigenvectors of X T X . It follows that β ^ PCR = U ( m ) T β ^ OLS .

PCR gives a biased estimate of the regression coefficients. If all the PCs are included in the model, we retrieve the usual MLR estimator, β ^ OLS .

As with PCR, PLS regression, introduced by [35] , consists in regressing y on components, also called latent variables, which are linear combinations of the p predictor variables.

The major difference between PCR and PLS regression is that, whereas PCR uses only X to construct the components to be used as regressors, PLS regression uses both X and y to determine such components. More precisely, the PLS components are determined sequentially and, at each step, we seek to determine a new component, constrained to be orthogonal to the components determined at the previous stages, so as to maximize the covariance between this component and the independent variable.

Suppose that m PLS components are determined. Again, the number m of latent variables to be introduced in the model can be selected by means of LOO cross validation technique in practice. These latent variables can be stacked into a matrix T ( m ) , T ( m ) = X W ( m ) where W ( m ) = ( w 1 , ⋯ , w m ) is the X-weight matrix. Equation (5) gives the vector of fitted values obtained by regressing y on T ( m ) , namely:

y ^ PLS m = T ( m ) ( T ( m ) T T ( m ) ) − 1 T ( m ) T y (5)

We can show in Equation (6) the expression of PLS regression coefficients as:

β ^ PLS = W ( m ) ( P ( m ) T W ( m ) ) − 1 ( T ( m ) T T ( m ) ) − 1 T ( m ) T y (6)

where P ( m ) = X T T ( m ) ( T ( m ) T T ( m ) ) − 1 and W ( m ) = X T y / ‖ X T y ‖ , where ‖   ‖ is the L² norm.

PLS regression is often helpful to reduce the number of predictors to a small number of latent variables constructed by linear combinations of the columns of original predictors. It yields a biased estimate of the regression coefficients.

The Sparse PLS method defined by [32] is a direct adaptation of the PLS regression method. It allows us to operate a dimensionality reduction using regression PLS.

In SPLS regression, w the first vector of loadings is sought as an optimal solution to:

max w ( w T M w ) subject to ‖ w T w ‖ = 1 , ‖ w ‖ 1 ≤ η ,

where M = X T y y T X , ‖ w ‖ 1 is the L¹-norm of vector w and η > 0 is a scalar which controls the degree of sparsity.

The regression coefficients estimation of y on X is calculated in the following way: The coefficients of the non-selected variables are set to 0, and the coefficients of the selected variables are those obtained by means of the “standard” PLS regression. We can also give an expression of the SPLS regression coefficients defined by (7) [32]

( β ^ SPLS ) j = { ( β ^ PLS ) j ,     if   w j ≠ 0     and     j = 1 , ⋯ , m 0                                       otherwise (7)

The interest of the SPLS is two folds. On the one hand, thanks to the sparsity, it yields an easy to interpret model and, on the other hand, it prevents the problem of multicollinearity by using the PLS framework. SPLS estimator is biased comparing to OLS estimator. Moreover, SPLS is computationally efficient with a tunable sparsity parameter to select the important variables.

The CR prediction model is chosen from a continuum of candidates among which we find methods of analysis related to OLS estimation, PCR, PLSR. [19] gives a general overview of the continuum approach regression and shows how different methods relate to “least squares ridge regression”. As with PCR and PLS regression, CR consists in a regression upon latent variables (i.e. optimal linear combinations of the independent variables). More precisely, these latent variables are determined in a sequential way where at each stage, a latent variable is defined so as to realize a balance between stability (as assessed by the variance of the latent variable) and prediction ability (as assessed by the correlation between the latent variable and the dependent variable y ). The prediction of y using CR, y ^ CR , is given by y ^ CR = X β ^ CR . The CR achieves a reduction of the variance of estimator at the cost of introducing a small bias [18].

CR aims at transforming the explanatory variables into new latent predictors which are orthogonal to each other and constructed as linear combinations of the original predictors. It makes it possible to circumvent the problem of multicollinearity between predictors. However, the CR regression does not specifically aim at selecting a subset of variables [18].

Another general strategy to circumvent the problem of multicollinearity consists in imposing a constraint on the vector of regression coefficients. The two most popular methods in this context are Ridge and Lasso regressions.

Ø Ridge regression

Ridge regression is the first regularization procedure that was proposed to cope with the multicollinearity problem [12]. The Ridge estimator is given by (8):

β ^ R = ( X T X + k I ) − 1 X T y (8)

where k ≥ 0 is a constant to be selected. Note that if k = 0 , the Ridge estimator amounts to the least-squares estimator.

Ridge estimator is obtained as a solution to the following least squares problem defined by (9):

β ^ R = arg min β ∈ ℝ p , ‖ β ‖ 2 ≤ δ ( ‖ y − X β ‖ 2 )     where   δ ≥ 0 (9)

There is a one to one correspondence between the Ridge parameter k and the upper bound, δ, imposed on the vector of regression coefficients, β . From a practical point of view, these parameters can be selected by means of a cross-validation technique.

The Ridge regression shrinks the OLS estimators towards 0. It yields a biased estimator, but with a smaller variance than that of OLS estimator.

Ø Lasso regression

The Least Absolute Shrinkage and Selection Operator, or Lasso [13] is another penalized regression where L² penalty of ridge regression is replaced by an L¹ penalty: ‖ β ‖ 1 = ∑ j = 1 p | β j | . This is a subtle change that has important consequences. Indeed, this constraint entails that some of the regression coefficients are shrunk exactly to zero. This means that this regression strategy operates de facto a selection of variables since the unimportant variables are discarded, their regression coefficients being equal to zero. Formally, the lasso estimator is given as a solution to the following optimization problem by (10):

β ^ Lasso = arg min β ∈ ℝ p , ‖ β ‖ 1 ≤ δ ( ‖ y − X β ‖ 2 ) , (10)

where δ ≥ 0 .

The parameter δ controls the degree of sparsity and, in practice; it is determined by Leave One Out (LOO) cross validation procedure. The smaller this parameter is, the larger is the number of discarded variable. Contrariwise, if δ is larger than δ 0 = ∑ j = 1 p | β ^ j | (where β ^ j are the OLS estimators) then β ^ Lasso = β ^ OLS . Lasso regression has the double effect of shrinking the β coefficients, allowing to decrease the variance of the regression coefficients as with Ridge regression, and, more importantly, to operate an automatic selection of the variables, by cancelling out some β j coefficients.

Recently, a new biased regression called Biased Power regression (BPR) strategy was proposed [20]. It consists in gradually shedding off the correlation among the independent variables by means of a tuning parameter α. More precisely, the BPR estimator of β is given by (11):

β ^ BP = ( X T X ) α − 1 X T y (11)

where α is a tuning parameter which ranges between 0 and 1.

In practice, α is selected using a cross validation procedure.

Clearly, when α = 0 , we retrieve the OLS estimator and as α increases, the variance-covariance matrix of the predictor variables is shrunk to the identity matrix. The prediction of y using BPR, y ^ BP is given by y ^ BP = X β ^ BP .

BP-regression shares the same properties as Ridge regression (see Section 2.3.4) and thus can highlight those variables whose coefficients become very small. However, it was not designed to select a subset of variables [20].

As mentioned above, a strategy based on the K-nearest neighbors was performed. Different K values were used in the literature and the choice of K = 10 led to the best results [38].

Figure 2 shows the scatter plots associated with pairs of the available variables. It highlights the pairwise correlations between these variables. We also indicate in Figure 2, the histograms associated with each variable and the correlation coefficients between each pair of variables. For instance, we can see that there is a high correlation between O3veilleJahid and O3 Jahid (the two last columns) with a correlation coefficient equal to 0.92. We can also observe large correlations (around 0.94) between the first three explanatory variables (TMPMAX and TMPMOY, TMPMIN and TMPMOY).

The diagonal entries show the histograms associated with the various variables and the upper entries indicate the coefficients of correlations between pairs of variables.

We also performed a PCA on the imputed dataset. PCA is run on the complete data (2013 and 2014) after imputation and standardization of the variables. The data is composed of 366 days (from April to September of 2013 and 2014) and 24 variables. The first five principal components recover up to 65% of the total variance (Table 2). In the following, only the results related to the first two principal components which recover around 40% of the total inertia are shown.

Figure 3 shows the correlations of the explanatory variables with the first two principal components. The variable O3 Jahid is superimposed as an illustrative variable with a blue arrow to depict its relationships with the explanatory variables. This figure highlights the strong correlations among the variables, which may be harmful for the prediction models.

The first PC is linked to wind direction (Vx06, Vx12, Vy12 and Vx18) and pressure variables (PRESTN at 06 h, 12 h and 18 h). We can notice, for example, that variables TMPMAX, TMPMIN and TMPMOY as well as PRESTN06h, PRESTN12h and PRESTN18h are strongly correlated. A strong correlation also exists between variables Vx06, Vy06, Vx18 and Vx12. O3veilleJahid variable is very correlated to O3 Jahid but it is not very well represented in the plan (PC1-PC2).