The MDL approach to model fitting can be described as follows (see [5,6]). Suppose we have some data. Consider a collection of probability models for this set of data. A model provides a better fit if it can provide a more compact description for the data. In terms of coding, this means that according to MDL, the best model is the one which provides the shortest description length for the given data. The MDL approach as discussed here is also related to the Minimum Message Length (MML) approach of [
If data 
is known to arise from a probability density
, then (see [
. (Here 
is logaritm to the base 2.) This is the link between description length and model fitting.
The optimal code length of 
is valid only in the discrete case. To handle the continuous case later, discretize x and denote it by 
where 
denotes the precision. In effect we will then be considering
instead of 
itself as far as coding of x is considered when x is one-dimensional. In the r-dimensional case, we will replace the density 
by the probability of the 
-dimensional cube of side 
containing
, namely
, so that the optimal code length changes to
.
, and suppose
be a prior density for
. Given a value of 
(or a model), the optimal code length for describing 
is
, but since 
is unknown, its description requires a further 
bits on average. Therefore the optimal code length is obtained upon minimizing
given the data 
is
is the marginal or predictive density. Therefore, minimizing
is equivalent to maximizing
. Thus MDL for estimation or model fitting is equivalent to finding the highest posterior density (HPD) estimate of
. Note, however, that a prior 
is needed for these calculations. The approach that a Bayesian adopts in specifying the prior is not, in general, what is accepted by practitioners of the MDL approach. Therefore, the equivalence of MDL and HPD approaches is either subject to accepting the same prior, or as an asymptotic or similar approximation. MDL mostly prefers an approximately uniform prior when 
for some fixed 
(same 
across all models), leading to the maximum likelihood estimate (MLE). The case of 
having model parameters of different dimensions is different and is interesting. This can be easily seen in the continuous case upon discretization. Now denote 
by 
and 
by
. Then
and 
are the precisions required to discretize 
and
, respectively. Note that the term 
is common across all models, so it can be ignored. However, the term 
which involves the dimension of 
in the model varies and is influential. According to [6,12], 
is optimal (see [
should depend on the Fisher information matrix. This amounts to using a prior which is similar in nature to the Jeffreys’ prior on
. Jeffreys prior is an objective choice and thus this approach to MDL can then be considered a default Bayesian approach.
, which is not the usual Bayes estimate. Posterior mean is what is generally preferred, so that the error in estimation has an immediate simple answer in the posterior standard deviation. In summary, therefore, the Bayesian approach doesn’t seem to find attractive solutions in the MDL approach as far as estimation or model fitting is concerned unless the models under consideration are hierarchical having parameters of varying dimension. On the other hand, when such hierarchical models are of interest the inference problem usually involves model selection in addition to model fitting. Thus the possible gains from studying the MDL approach are in the context of model selection as described below.
. Suppose
. Consider testing
, for some
and
. Let 
be a prior on
. Then 
can be expressed as
and 
and 
are the conditional densities (with respect to some dominating 
- finite measure) of 
under 
and 
respectively. Then
is simply the marginal or predictive density of 
under 
and 
is the unconditional predictive density obtained upon averaging 
and
. Consequently, the posterior odds ratio of 
relative to 
is,
denoting the Bayes factor of 
relative to
. When we compare two competing models 
and
, we usually take
, and hence settle upon the Bayes factor 
as the model selection tool. This agrees well with the intuitive notion that the model yielding a better predictive ability must be a better model for the given data.
and
. Once the conditional prior densities 
and 
are agreed upon, MDL will select that model 
which obtains a smaller value for the code length, 
, between the two. This is clearly equivalent to using Bayes factor as the model selection tool, and hence this version of MDL is equivalent to the Bayes factor criterion. In the MDL literature, this version of MDL is known as “mixture MDL”, and is distinguished from the “two-stage MDL” which separately codes the model and the prior. The two-stage MDL can be derived as an approximation to the mixture MDL as discussed later. See [
with known
. We want to test
prior on 
with known 
under
. Then the marginal distribution of 
is
under 
and under 
it is
. A continuous model and a continuous prior is considered here. Since the precision of the prior parameter is the same across all models upon discretization we will ignore the distinction and proceed with densities. Then, both the Bayes factor criterion and the MDL principle will select 
over 
if and only if
and 
are the corresponding densities. Since we are comparing two logarithms, let us switch to natural logarithms. Then
and
is preferred over 
either by the Bayes factor or by the mixture MDL if and only if
now. Suppose the prior on 
is the default 
under both models. The prior on 
under 
is now assumed to depend on
, i.e., 
, where 
is assumed to be a known constant for now. Then, provided
,
, and 
denote the marginal density of 
given 
and
,
is now replaced by an estimator.
, with density
is the same as before under both models:
. This approach suggested by Jeffreys ([
is the same as in the previous example, namely
can be expressed as a Gamma scale mixture of normals,
is the mixing Gamma variable. Now one can integrate over 
and 
in closed form to simplify (m1xn). Finally, one has a one-dimensional integral over 
left, which can be numerically computed whenever needed.
.
and 
are smooth functions, we obtain, for large
, the following asymptotic approximation for 
of equation (8). Let 
be the dimension of
,
and 
denote the Hessian of
, i.e.,
denote either the MLE or the posterior mode. Then
, [
are i.i.d. observations, then we have
is the Fisher Information matrix and hence
, we obtain the Schwarz criterion ([
given by
and
. Model selection criterion of Rissanen is SIC as described in Equations (13) and (14). However, in this problem a better approximation is employed for the mixture MDL, the mixture being Jeffreys mixture, i.e., the conditional prior densities under 
for 
are given by
is a compact subset of the relevant parameter space 
and
. Consequentlyit follows that,
is the MLE of 
under
. This yields,
vanishes.
where its integral is finite:
with the HPD estimate when there is an informative prior, or with the MLE, but the resulting minimum description length does have interesting features. To see when and how this approach approximates the above mentioned model selection criterion, let us look at some of the specific details in the two stages of coding. See [12,13] for further details. Again, recall the setup in (4) and (5).
be an estimate of 
such as the posterior mean, HPD or MLE under
. This needs to be coded. Consider the prior density 
conditional on 
being true. Usually MDL would choose a uniform density. Restrict 
to a large compact subset of the parameter space and discretize it as discussed in Section 3 with a precision of
. Then the codelength required for coding 
is
is coded using the model density
. Discretization may again be neededsay with precision
. Thus the description length for coding 
will be
, is constant over both M0 and
, and the third term stays bounded as 
increases, these two terms are dropped from the MDL two-stage coding criterion. Thus, for regular parametric models, the two-stage MDL simplifies to the same criterion (for
) as BIC, namely,
since SIC uses a better precision for coding the parameter, namely, one based on the Fisher information.
on a set of potential explanatory variables (or regressors). Following [
, a binary variable
. Then the set of available linear models is
. Note that 
is, then, a Bernoulli sequence associated with the set of regression coefficients, 
also. Let 
denote the vector of non-zero regression coefficients corresponding to
, and 
the corresponding design matrix, which results in the model
starting with a prior 
on 
and a prior 
on 
given
. The two-stage MDL, which is the simplest, uses the criterion of minimizing
and 
given 
are easily available:
, all 
values receiving the same weight
. Using these, we can re-write the MDL criterion as the one which minimizes (as in Example 2)
is the number of
.
. If 
is the prior density under
,
and ignoring terms that are irrelevant for model selection, one obtains (see [
is chosen to be the conjugate prior density, then the marginal density 
can be explicitly derived. Details on this and further simplifications obtained upon using Zellner’s g-prior can be found in [
models is not a computationally viable option for even moderate values of
, since for each model, 
, one has to compute the corresponding 
and
.
, and hence 
is the intercept which gives the starting model. Then we decide whether this model needs to be expanded by adding additional regressors. Thus, at step
, we have an existing model with regressors 
and we fix 
to be one of the remaining 
regressors as the candidate for possible selection. Now the two-stage MDL approach is straight forward. From (22) and (24), we note that 
is to be selected if and only if
is the description length of the model with regressors 
and 
is its residual sum of squares as given in (23). A closer look at (27) reveals certain interesting facts. We need the following additional notations involving design matrices and the corresponding projection matrices. We assume that the required matrix inverses exist.
is simply the partial correlation coefficient between 
and 
conditional on
. Substituting these in (27), we see that
if and only if
if and only if
and
, respectively, are the prior densities under 
and
. Taking these priors to be g-priors, namely,
for
, a (proper prior) density 
for the hyperparameter
, we obtain,
and
.
with
, where 
are the ML-II (cf. [
. See [
, we consider only those regressors (which are not already in the model and)
(=0.1005 in this example). From this set we pick the one with the largest
. The values of 
for the relevant steps are listed below.
first, followed by 
and the selection ends there.
, we pick the regressor 
with the largest value for
for the relevant steps are given below.
.
:
are i.i.d. 
errors with unknown error variance
, and 
is a function (or signal) defined on some interval
. Assuming s is a smooth function satisfying certain regularities (see [15,39,40]), we have the wavelet decomposition of
:
are the wavelet functions and
is the corresponding vector of wavelet coefficients. We assume that the normally infinite sum in (36) can be taken to be a finite sum (or at least a very good approximation) as indicated in [
, we get the estimated wavelet coefficients,
. Consider now the equivalent model:
.
are the number of wavelet coefficients of interest.
is assumed to be i.i.d. 
under
.
, we proceed as follows to describe the prior structure. With each 
we associate a binary variable 
as in [
is a Bernoulli sequence associated with the set of regression coefficients,
. Let
be i.i.d. 
(
be the corresponding joint density), and define the following structure under
.
which is common under both models is given the prior density
. Then it follows that
for which 
appear in the integral above.
by coding the prior and the likelihood (both evaluated at an estimate) separately and sums the codelengths to obtain the description length. In this case, discretizing 
and 
to a precision of 
and ignoring terms that stay bounded as 
increases, this amounts to
is for coding the 
non-zero
’s and 
is an estimate such as the MLE. On the other hand, computing SIC or 
is not an impossible task either. In fact, to integrate out the 
in 
of Equation (40) we argue as follows.
and
, and integrate out 
also. This leaves us with the following expression where we have a sum over
.
is interesting. Most of the contribution to this sum is expected from 
with 
corresponding to the largest 
of the
, which yields the MLE of 
upon normalization. The Bayes estimate, on the other hand, will arise from a weighted average of all the sums, with weights depending on the posterior probabilities of the corresponding
. As is clear, weighted average over the space 
is computationally very intensive when 
and 
are large. An appropriate approximation is indeed necessary, and MDL is important in that sense.
are i.i.d. 
is unreasonable; wavelet coefficients corresponding to wavelets at different levels of resolution must be modeled with different variances. Specifically they should decrease as the resolution level increases to indicate their decreasing importance (see [40,42,43]). Secondly, wavelet coefficients tend to cluster according to resolution levels (see [
be independent Poisson counts with
. 
are a priori considered related, and a joint multivariate normal prior distribution on their logarithm is assumed. Specifically, let 
be the 
th element of 
and suppose
is the possible change-point. We further let 
be i.i.d.
. Note that 
and 
are hyperparameters.
as follows.
. Expanding (43) about
, its maximum, in Taylor series and ignoring higher order terms in
, we obtain
) with mean vector x and covariance matrix 
where
and
and 
do not have substantial influence and hence treat them as fixed constants (to be chosen based on some sensitivity analysis) in the following discussion. Consequently, denoting by 
and 
the respective prior densities under 
and
,
can be integrated out as in Example 2. Thus,
. Again, from (47), an ML-II likelihood for 
is given by 
which maximizes
and 
this involves only examining a smooth function of a single variable, 
, which is a simple computational task.
also. Partition the required vectors and matrices as follows.
from
, for fixed 
and
. Obtaining 
(for fixed 
and
) which maximizes 
is very similar to that for
.
and 
as a function of 
for 
and
, for the LGV and HGV data, respectively. As mentioned previously, 
and 
do not seem to play any influential role; any reasonably small value of 
seems to yield similar results, and any 
which is not too close to 0 behaves similarly.
near the ML-II estimate of 
for the LGV data.