The paper introduces a new Frequentist model averaging estimation procedure, based on a stacked OLS estimator across models, implementable on cross-sectional, panel, as well as time series data. The proposed estimator shows the same optimal properties of the OLS estimator under the usual set of assumptions concerning the population regression model. Relatively to available alternative approaches, it has the advantage of performing model averaging exante in a single step, optimally selecting models’ weight according to the MSE metric, i.e. by minimizing the squared Euclidean distance between actual and predicted value vectors. Moreover, it is straightforward to implement, only requiring the estimation of a single OLS augmented regression. By exploiting exante a broader information set and benefiting of more degrees of freedom, the proposed approach yields more accurate and (relatively) more efficient estimation than available expost methods.
Model Averaging; Model Uncertainty1. Introduction
The Classical Linear Regression Model (CLRM) is grounded on a basic set of assumptions concerning its specification and distributional properties of control variables and error term. In this respect, under what is usually held as Assumption 1, the population regression model is required to be linear in the parameters, and control variables are all known and included in the model. However, the latter correct specification assumption may not always be appropriate in Economics; for instance, there may be more than a single set of variables, i.e. more than a single candidate model, which can be employed in estimation, also when economic theory has clear-cut implications for the causal linkage of interest.
Consider the relationship linking y to x, when both variables can be measured in different ways, i.e. when there exist and, ,; then, in principle, up to, different models can be estimated.1
Two solutions have so far been proposed in the literature to the above model selection problem. On the one hand, by maintaining the assumption of correct specification, a single model out of the candidates can be selected according to various specification strategies (see [2] for a general account; see also [3] for recent developments in model selection). Alternatively, all of the models can be estimated, and a weighted average across models computed ex-post for the parameters of interest. In the latter case, the assumption of correct specification does not have necessarily to be maintained.
Several model averaging procedures have been proposed in the literature, making use of either Bayesian or Frequentist procedures (see [4] [5] ). Admittedly, relatively to Bayesian, the Frequentist approach to model averaging is fairly underdeveloped. The current paper then aims at filling this gap in the literature, by proposing an ex-ante, mean square error-optimal model averaging procedure. The proposed procedure is grounded on a stacked OLS estimator across models, implementing model averaging ex-ante in a single step, optimally selecting models’ weight according to the MSE metric, i.e. by minimizing the squared Euclidean distance between actual and predicted value vectors. Moreover, it is straightforward to compute, only requiring the estimation of a single OLS augmented regression. By exploiting a broader information set ex-ante, i.e. by making use of all the available information jointly, and benefiting of more degrees of freedom, the proposed estimator then yields more accurate and (relatively) more efficient estimation than available ex-post methods. Extension to other estimation frameworks, i.e. GIVE or GMM, is also straightforward.
The rest of the paper is organized as follows. In Section 2, the proposed approach is illustrated by means of a simple example. Then, the econometric methodology is outlined in full in Section 3, while Section 4 deals with its statistical properties. Finally Section 5 concludes.
2. Ex-Ante Model Averaging: An Example
For sake of clarity, consider the following bivariate example
where the dependent variable y is a linear function of the independent variable x. The endogenous variable y can then be alternatively measured by and, while the independent variable x by and. In what fol-
lows we assume that the other usual properties of the CLRM hold, i.e., , ,
, is a stationary and ergodic process, of zero mean for simplicity; the regressors and the residuals
are at least contemporaneously orthogonal, i.e.; the residuals are conditionally homoskedastic and non serially correlated,).2
Four consistent estimates of the parameter of interest are then obtained, i.e., , , , by means of OLS estimation of each of the four available alternative models
Ex-post model averaging then yields a robust consistent estimate of, by computing a weighted average of the four available estimates, , , , with weights determined according to Bayesian or Frequentist approaches.
For instance, within a Frequentist model averaging approach [2] , one has
where the weights can be computed by means of information criteria as in [6] , setting
where is the Akaike or Schwarz-Bayes information criterion for model. Other approaches are also available, based on Mallow's criterion [7] or cross-validation [8] .
On the other hand, the proposed model averaging strategy is single-step and implemented by means of an augmented regression model using all the available data jointly. It then requires the construction of the auxiliary dependent () and independent () variables, by appropriately stacking the actual data and in single column vectors.
With reference to the set of models in (2), consider the stacked model obtained from their union, i.e.
where, and are vectors,
;, , , , are vectors containing the observations on, and, respectively.
Alternatively, the regression model can be written as
The stacked OLS problem is then stated as
yielding, after some algebra
or
where
with.
The ex-ante model averaging or stacked OLS estimator of is then equivalent to its ex-post counterpart, with weights determined according to the relative variation of the candidate regressors.
Moreover, consistent OLS estimation of from the generic th disjoint model yields
while the stacked estimator is
Hence, the stacked OLS estimator of is equivalent to the arithmetic mean, across models, of the disjoint OLS estimators of.
Issues related to the (relative) efficiency of the stacked OLS estimator and the gain in terms of higher degrees of freedom are discussed below.
3. Ex-Ante Model Averaging by Stacking
Consider the regression function
and suppose that P candidate dependent variables are available, i.e., where, , is a column vector of observations.
For simplicity, three cases for the specification of the design matrix are considered:
1) The case of a single design matrix for the K regressors, , where is a vector and.
2) The case of R candidates for one of the K regressors in the model, ordered first for simplicity, i.e., , , yielding up to R different design matrices.
3) The case of R candidates for each of the K regressors in the model, yielding up to different design matrices,.
3.1. The Case of a Single Design Matrix
In case 1. Up to P models could be estimated, i.e.
Their union yields the stacked model
where is a vector of observations on the P available candidate dependent variables, obtained by stacking the P column vectors on top of one other; is the joint design matrix obtained by staking P times the matrix on top of itself, i.e., is the vector of parameters, and is the vector of residuals, obtained by stacking the P column vectors on top of one other. Hence, the sample size of the stacked model is.
Disjoint OLS estimation of the pth generic model in (15) yields (see [9] )
while for the variance, in large samples
The Ex-Ante Model Averaging Estimator
Ex-ante model averaging is obtained by OLS estimation of the stacked model in (16), yielding
The linkage between ex-ante and ex-post model averaging can then be gauged by noting that (19) can be stated as
where,.
Hence, in this case, ex-ante OLS model averaging is equivalent to ex-post arithmetic model averaging across the P disjoint OLS estimators.
Similarly for
which also is the arithmetic average, across the P available models, of the disjoint estimators.
3.2. The Case of Multiple Design Matrices
In the case of multiple design matrices, up to G regression models can be computed, with in case 2. and in case 3., i.e.
The disjoint OLS estimator for the generic th model, , , in (23)
is
while for the variance, in large samples
On the other hand, the union of the above disjoint models yields the stacked model
where is the vector of parameters, is the vec-
tor collecting the P vectors, , which are then stacked on top of one other G times, is the vectorization operator, is the Kronecker product and a unitary vector.3
By denoting the matrix obtained by stacking the G candidate design matrices on top of one another, is then the design matrix yield by staking P times the matrix on top of itself, i.e.. Finally, is a vector of residuals. Hence, the sample size of the stacked model is.
The stacked OLS estimator is then computed as
3.2.1. The Case of a Single Candidate Dependent Variable
For sake of simplicity, consider first the case where; hence, , , and the design matrix in the stacked model is
The stacked OLS estimator in (28) can then be stated
where
Denote, yielding, , and so on. By substitution in (30), it follows
Using matrix inversion rules4, one has
where
By substitution in (31), it follows
where.
Optimal ex-ante weights, contained in the matrices, , are then computed by taking into account all the information available on the various candidate regressors, being proportional to their relative variation. In fact, multiplying both sides of (32) by, one has
and therefore.
Moreover, given, one has
Hence, is the arithmetic average, across the available G models, of the disjoint estimators.
3.2.2. The Case of Multiple Candidate Dependent Variables
Consider now the case in which more than single candidate dependent variable is available, i.e.. The stacked OLS estimator in (28) is then
where again.
Moreover, denote, i.e., , and so on; by substitution in (34), one then has
By recalling that, where, by substitution in (35) one eventually has
where, as for the previous case,.
The optimal ex-ante weights, contained in the matrices, , are again computed by taking into account all the information available on the various candidate regressors and are proportional to their relative variation. Averaging is then performed across all possible models which can be estimated according to the P candidate dependent variables.
Moreover,
Then, ex-ante model averaging estimation of the variance is computed as the arithmetic average, across all the models, of the disjoint estimators.
4. Statistical Properties
Assume that the properties of the classical linear regression model hold, i.e.:
1) The population regression function is linear in the K parameters, i.e..
2) is a candidate stationary and ergodic process,;,;
is a vector of regressors (belonging to the rth design matrix) at observation t,;.
3) The regressors are at least contemporaneously orthogonal to the residuals, i.e., where is the residual from the generic prth model at observation t.
4) Any of the design matrices has rank equal to K with probability 1, with a finite, symmetric, invertible, positive semidefinite matrix.
5) The conditional variance covariance matrix of the residuals is a scalar identity matrix, i.e.
, implying that the residuals are conditionally homoskedastic and non serially correlated,).
Under the above assumptions (even relaxing the conditional homoskedasticity property), the disjoint OLS estimator in (25) and in (26) is consistent and asymptotically normal (see [9] ). The same properties hold for the stacked OLS estimator. Proofs for the most general case are reported below; results for the intermediate cases can be straightforwardly derived from those provided, by setting or.
4.1. Large Sample Properties
In so far as, it follows for in (36)
since by ergodic stationarity, where is a finite and non singular matrix and
.
Moreover, in so far as, it follows for in (37)
Under properties 1. to 5., by means of a CLT (see [9] ), one also has
leading to
The asymptotic distribution of then follows
as well as its feasible form
In the case of conditional heteroskedasticity, it would be straightforward to prove that
and
with feasible form
where.
The relative efficiency of the stacked over the disjoint OLS estimator can be established by comparing their asymptotic variances, i.e. and. One then has
which is a finite, symmetric, positive semidefinite matrix, as and, both finite, and
is a finite, symmetric, positive semidefinite matrix by construction (is real and of full column rank for any r).
Finally, the gain in terms of degrees of freedom yield by the stacked over the disjoint OLS estimator is equal to. In fact, by recalling that the number of degrees of freedom of the residual term is equal to the rank of the annihilator matrix (see [9] ), the gain yield by stacked over disjoint OLS estimation can be established by comparing the rank of the annihilator matrix in the two cases
which is of rank equal to as
and
which is of rank as
The increase in degrees of freedom yield by stacked over disjoint OLS estimation is then
4.2. Small Sample Properties
If the stronger assumption of strict exogeneity is made in 3. above, i.e., the disjoint OLS estimators in (25) and are also (conditionally and unconditionally) BLUE, i.e. best un-
biased and efficient (within the class of linear estimators) (see [9] ).5 Moreover, if the assumption of conditional Normality of the error term is added, i.e., OLS is (conditionally and unconditionally) BUE, i.e. best unbiased (within the class of linear and non linear estimators), as well as (conditionally and unconditionally) normally distributed
where is a finite, nonsingular, symmetric, positive semidefinite matrix of rank.
The above properties can also be established for the stacked OLS estimator, in the same way as for the disjoint OLS estimator (see [9] ), yielding
with a finite, nonsingular, symmetric, positive semidefinite matrix of rank, and feasible form
where.
Then, by comparing the conditional variances of and, one has again
as for the asymptotic case. Moreover,
which similarly is a finite, symmetric and positive semidefinite matrix by construction.
Finally, the gain in terms of degrees of freedom yield by stacked over disjoint OLS estimation is again, as already shown for the asymptotic case.
5. Conclusion
The paper introduces an ex-ante model averaging approach, requiring the estimation of a single augmented model obtained from the union of all the possible candidate models, rather than their disjoint estimation. In this framework, optimal weights are implicitly computed according to the MSE metric, i.e. by minimizing the squared Euclidean distance between actual and predicted value vectors, and are proportional to the relative variation of the regressors. By exploiting ex-ante all the available information on the various candidate set of variables, and relying on more degrees of freedom, it then leads to more accurate and (relatively) more efficient estimation than available ex-post methods. Moreover, the proposed estimator shows the same optimal properties of the disjoint OLS estimator, under the usual set of assumptions concerning the population regression model. While the method is proposed to be used within the OLS estimator framework, extension to GIVE and GMM is straightforward. We point to [1] for an empirical application using OLS and GMM estimation.
Acknowledgements
The author is grateful to the referees for their comments. This project has received funding from the European Union’s Seventh Framework Programme for research, technological development and demonstration under grant agreement no. 3202782013-2015. The flowers are supported by the branches/The trunk supports the branches/The roots support the trunk/But we do not see the roots (Mitsuo Aida).
Cite this paper
ClaudioMorana,11, (2015) Model Averaging by Stacking. Open Journal of Statistics,05,797-807. doi: 10.4236/ojs.2015.57079
NOTESReferencesBaiardi, D. and Morana, C. Financial Deepening and Income Distribution Inequality in the Euro Area. Department of Economics, Management and Statistics, University of Milan Bicocca, Working Paper No. 316.Claeskens, G. and Hjort, N. (2008) Model Selection and Model Averaging. Cambridge University Press, Cambridge. </br>http://dx.doi.org/10.1017/CBO9780511790485Hendry, D.F. and Doornik, J.A. (2004) Empirical Model Discovery and Theory Evaluation. MIT Press, Cambridge.Hoeting, J.A., Madigan, D., Raftery, A.E. and Volinsky, C.T. (1999) Bayesian Model Averaging: A Tutorial. Statistical Science, 14, 382-417.Moral Benito, E. (2015) Model Averaging in Economics: An Overview. Journal of Economic Surveys, 29, 46-75. </br>http://dx.doi.org/10.1111/joes.12044Buckland, S., Burnham, K. and Augustin, N. (1997) Model Selection: An Integral Part of Inference. Biometrics, 53, 603-618. </br>http://dx.doi.org/10.2307/2533961Hansen, B. (2007) Least Squares Model Averaging. Econometrica, 75, 1175-1180. </br>http://dx.doi.org/10.1111/j.1468-0262.2007.00785.xHansen, B. and Rancine, J. (2010) Jacknife Model Averaging. Journal of Econometrics, 167, 38-46. </br>http://dx.doi.org/10.1016/j.jeconom.2011.06.019Hayashi, F. (2000) Econometrics. Princeton University Press, Princeton.