After the within subject correlation is taken into account in general linear models, the random errors are not assumed independent. Linear mixed models are setting up to solve this problems. Previous literature work on linear models and linear mixed models has mainly focused on setting up a list of possible covariance matrix structures of random errors and random effects. However, a potential vital risk still exists. If no one structure in list can fit data properly, even the best one in that list may distort statistical inference. Misspecification of covariance structures can lead to a great loss of efficiency of the parameter estimates (Wang and Carey, 2003, [1] ).

Joint modelling (Pourahmadi, (1999), [2] ) is a data-driven methodology for modelling the variance-covariance matrix similar to the general linear model framework for mean modelling. The main idea of joint modelling is considering mean and variance components in equally important level, modelling the mean, variance and correlation of response variables in three submodels, building three design matrices for mean, variance and correlation respectively, and estimating three parameters vectors for these three submodels simultaneously. The advantage of these method is to reduce the high dimensionality of computation for the covariance matrix. Modified Cholesky decomposition of the inverse of covariance matrix, , guarantees that and must be positive definite. The resulting parameters estimates are obtained and compared to those obtained using the above approaches. Assumed structures and joint modelling are compared and discussed further in terms of their ability to accurately estimate both regression coefficients and variance components.

In this paper, we aim to model covariance structures of random errors in the linear models in Section 2, model covariance structures of random errors and random effects in linear mixed models in Section 3 and model mean and covariance simultaneously based on modified Cholesky decomposition (Pourahmadi, (2000), [3] ) in Section 4. Cattle data published by Kenward in 1987 [4] were analyzed using the above three models in Section 5. The core conclusion is that real data analysis confirms the superiority of joint modelling of mean-covariance. Further discussions are followed in Section 6.

The linear regression model assumes that the response variables and the explanatory variables are related through

where, for the subject of of m subjects, is an responses, X_i is an design matrix, is a fixed effect, random error is an random vector and assumed as normal distributed with mean zero and covariance matrix identity structure, is an identity matrix. Under Gauss assumption, ,

However, response in real data may be not independent. The inferences of the regression parameters of primary interest take account of the likely correlation in the data. So linear models with correlated errors is considered,

Under this assumption, , where.

The distributional assumption about in Model (2) may be too restrictive. For example, when data are collected over time on an individual such as in repeated measurements data, it is more likely that observed measurements have a high correlation within individual. The linear mixed model extends the general linear model by allowing a more flexible specification of the covariance matrix of. In other word, it allows for both correlation and heterogenous variances, although we still assume normality.

Laird and Ware (1982) [5] provided the following description of a linear mixed model

where, for the subject of of m subjects, is an responses, is an design matrix, is a fixed effect. is an design matrix, is a random effect, , random effects are independent between subjects. Random error is an random vector and assumed as normal distributed with mean zero and covariance matrix identity structure , is an identity matrix., random errors are independent between subjects.

The model also can be written in

where Y is an responses, X is an design matrix, is a fixed effect, Z is an design matrix, is a random effect, is an random error.

The method of restricted maximum likelihood, or REML estimation, was introduced by Patterson and Thompson (1971) [6] as a way of estimating variance components in a general linear model. The objection to the standard maximum likelihood procedure is that it produces biased estimators. In REML, the degree of freedom of fixed effects are taken into account implicitly, unlike in the ML where they are not. That is why the ML estimator are biased.

REML is an alternative maximum likelihood procedure which involves applying the ML idea to linear combinations of y. Denote and B the matrix defined by the requirements that and, where I denote the identity matrix. Finally, let.

For fixed, where is variance components parameters, defined as in Section 2, the maximum likelihood estimator for is

Also, the respective pdf of Y and are

and

Furthermore, and Z and are independent, whatever the value of, as we now show.

Firstly, we prove,

since. But since, this gives

Hence as required.

Secondly, we prove

Now, substituting into this last expression gives

Also,

and

Where as in the proof that.

It follows that. Finally, in the multivariate Gaussian setting, zero covariance is equivalent to independence. It follows that the algebraic form of the (singular) multivariate Gaussian pdf

of Z, expressed in term of Y, is proportional to the ratio. To obtain the explicit form of this ratio, we use

the following standard result for the general linear mixed model that

Then, the pdf of is proportional to

where the omitted constant of proportionality is the Jacobian of the transformation from Y to. Harville

(1974) [7] shows that the Jacobian reduces to, which does not depend on any of the parameters in the

model and can therefore be ignored for making inferences about and. Note that the right hand side of (21) is independent of A, and the same result would therefore hold for any Z such that and.

The practical implication of (21) is that the REML estimator, , maximizes the log likelihood

whereas the maximum likelihood estimator, MLE, maximizes

Response variables was assumed Gaussian distribution

Kenward (1987) [4] reported a set of data on cattle’s growth. 60 cattle were randomly allocated into two groups, each of which would hold 30 animals. The animals were put out to pasture at the start of the grazing season, and the members of each group received one of two different treatments, A and B say, respectively. The weight (response) of each animal was then recorded 11 times to the nearest kilogram over a 133-day period. The first 10 measurements were made at two-week intervals and the final measurement was made after one-week interval.

We extract the best structure in different models from different models and put them in the Table 1. The second column gives the result for classical linear model with identical matrix of random errors in Section 2. The third column shows the result for linear model with AR(1) structure covariance matrix of random errors in linear model in Section 2. The fourth column represent the Model (13) (AR(1) structure for random effect and AR(1) structure for random error) in Section 3. The last column lists the joint modelling of mean-covariance regression in Section 4.

The main conclusions of this paper include:

a) The estimated mean value for the response variable and the estimators of fixed effects parameters are similar whatever the covariance structure is chosen.

b) The standard deviation of fixed effects parameters estimates is relatively different, which indicates the confidence intervals depend on the specification of covariance structure.

c) As a model selection criterion, AIC or BIC value is very important. The smaller the AIC or BIC value is, the better the model is. We conclude that the joint mean-covariance regression model is the best model for analysing cattle data. Note that in all of these models, the sample size in AIC or BIC value is 660.

Apart from the above three conclusions, we can see some other hiding connotations. In the first column of Table 1, we assume all measurements are independent, and then the maximum marginal log-likelihood value is −2782.644. When extending this model and assuming some correlations in successive measurements within individual, for example, the first order autoregressive model AR(1), the result shown in the second column becomes much better. The maximum marginal log-likelihood value increase from −2782.644 to −2289.697.

Simultaneously, AIC value reduces from 5575.288 to 4591.395 and BIC from 5597.749 to 4618.348. Obviously, these two values reduce remarkably. Large reduction implies that the correlation between successive observations should not be ignored. It also implies that the joint mean-covariance model provides correct information from the increasing of log likelihood or from changes of AIC and BIC value.

What’s more, we expect that if the structure of variance is AR(2), the result will be better than AR(1) structure. This part of work can be done as further work.

When we separate the variance-covariance matrix for each animal, , into two parts, for random effect between subjects and for random error within subject. The maximum log-likelihood value increases very little (from −2289.697 to −2273.92) compared to the last increase from column 2 to column 3. If is not AR(1) structure, for example, we try identity and identity for, the log-likelihood value in the linear mixed model is not better than that in general linear model, which indicates that the AR(1) structure for is more appropriate than other specifications.

Another further work on the cattle data analysis that may be carried out is to assume that different groups of animals may have different covariance structures, unlike in this dissertation where we assume the same covariance structure applied in both groups of animals. Other modelling methods, for example, nonlinear regression model might be also used to analyse cattle data.

We thank the editors and the reviewers for their comments. This research is funded by the National Social Science Foundation No. 12CGL077, National Science Foundation granted No. 71201029, No. 71303045 and No. 11561071. This support is greatly appreciated.

Yin Chen,Yu Fei,Jianxin Pan, (2015) The Effects of Covariance Structures on Modelling of Longitudinal Data. Open Access Library Journal,02,1-10. doi: 10.4236/oalib.1102086