First, let’s remember that a model is said to be linear if it is linear with respect to its parameters [4] . This means it can be represented either by a straight line or by a curve. This is the case, for example, with a polynomial model, which is classified as a linear model. In this case, the general form of a linear model is given by the following formula:

$g\left(z,w\right)={\displaystyle \underset{i=1}{\overset{p}{\sum }}}{w}_i{f}_i\left(z\right)$ (1)

where the functions $f_i\left(z\right)$ are non-parametric functions of the variables. We can replace them with $x_i$ and we obtain the form:

$g\left(x\mathrm{,}w\right)=w\cdot x$ (2)

In the formula (2), w represents the parameter vector and x represents a row of data. If we have n rows of data, we can define the observation matrix X, which will have n rows and p columns. Thus, the linear model becomes:

$g\left(X\mathrm{,}w\right)=X\cdot w$ (3)

Notation Given the data matrix X and y vector of data representing the phenomenon to be modeled, the problem therefore comes down to estimating the parameters w of the linear model of the g function. Least squares proposes the function to be minimized

$J\left(w\right)={\displaystyle \underset{k=1}{\overset{N_A}{\sum }}}{\left(y_k^p-g\left(x_k,w\right)\right)}^2$

où $N_A$ is the data sample size, $x_k$ is the vector of variables for the data k, and $y_k^p$ is the value of the quantity to be modeled for the data k. To find parameter values w for which this function is minimal, it suffices to write that its first derivative is zero:

${\nabla }_{w}J=\frac{\partial J\left(w\right)}{dw}=0$

which take a set of p equations, whose p unknowns are the parameters $w_i$, $i=1$? p. These equations are called normal equations. We show that

$w_{mc}={\left(X^{\text{T}}X\right)}^{-1}{X}^{\text{T}}{y}^p$ (4)

When X has a large dimension, recursive least squares is used [8] . The numerical method to be used can be either singular value separation or gradient descent [9] .

This solution can be obtained using a geometric method, which is an elegant interpretation of the method of least squares [10] . Determining the w coefficients of formula 3 is equivalent to solving the system $X\cdot w=y$. This system is incompatible because it has more equations than unknowns. Since a solution cannot be found because $y\notin ImX$, we’re forced to settle for finding a w’ such that $X\cdot w^{\prime }=y^{\prime }$ with y’ as close as possible to y. This means that $\forall {y^{\prime }}^{\prime }\mathrm{,}\exists y^{\prime }$ of $ImX$, $\Vert X{w}^{\prime }-{y^{\prime }}^{\prime }\Vert >\Vert X{w}^{\prime }-y^{\prime }\Vert$. From a geometric point of view, y’ closest to y is that which is the orthogonal project of y in the image of the matrix X. Therefore, all column vectors of X are orthogonal to the vector $y-y^{\prime }$. Thus,

$X^{\text{T}}\left(y-y^{\prime }\right)=0$

$X^{\text{T}}\left(X{w}^{\prime }-y^{\prime }\right)=0$

$X^{\text{T}}X{w}^{\prime }-X^{\text{T}}{y}^{\prime }=0$

$X^{\text{T}}X{w}^{\prime }=X^{\text{T}}{y}^{\prime }$

$w^{\prime }={\left(X^{\text{T}}X\right)}^{-1}{X}^{\text{T}}y$ (5)

We note that the solution of the formula 4 is the same as that obtained by the formula 5.

However, most of the time, researchers manipulate very limited data in the form of a sample, tainted by noise. The fact that a sample is drawn at random means that the solution may vary with respect to the sample drawn. Hence the need to formulate the problem of estimating the parameters of a model from a probabilistic point of view.

The frequentist method for estimating model parameters is maximum likelihood estimation [12] . It seeks to find the parameter values that make the observed data more probable within the framework of the model under consideration.

Thus the probability density of y knowing x is given by:

Let ${\epsilon }_k$ be the deviation between the predicted value of the model for data k in the sample and the observed value of the measurement to be modeled $y_k$ i.e. ${\epsilon }_k=y_k-g\left(w\mathrm{,}x\right)$. Suppose that ${\epsilon }_k$ follows a normal distribution with mean 0 and variance ${\delta }^2$ i.e. $\epsilon ~N\left(\mathrm{0,}{\delta }^2\right)$ et $\epsilon$ are independent.

This implies that y knowing x follows a normal distribution with mean $g\left(x\mathrm{,}w\right)$ and variance ${\sigma }^2$.

$P\left(y/x\mathrm{,}w\mathrm{,}{\sigma }^2\right)=N\left(g\left(x\mathrm{,}w\right)\mathrm{,}{\sigma }^2\right)=\frac{1}{\sqrt{2\pi {\sigma }^2}}\exp \left[-\frac{{\left(y-g\left(x\mathrm{,}w\right)\right)}^2}{2{\sigma }^2}\right]$ (6)

The maximum likelihood function L for n independent observations following a normal distribution of $\mu =g\left(w\mathrm{,}x\right)$ with variance ${\sigma }^2$ will be written as the product of the individual probability density of each observation.

$L\left(y/X\mathrm{,}w\mathrm{,}{\sigma }^2\right)={\displaystyle \underset{k=1}{\overset{N}{\prod }}}\left[N\left(g\left(x_k\mathrm{,}w\right)\mathrm{,}{\sigma }^2\right)\right]$ (7)

To simplify the calculation, we introduce the logarithm into the likelihood function, and obtain:

$\begin{array}{c}\ln L\left(y/X,w,{\sigma }^2\right)=\ln {\displaystyle \underset{k=1}{\overset{N}{\prod }}}\left[N\left(g\left(x_k,w\right),{\delta }^2\right)\right]\\ ={\displaystyle \underset{k=1}{\overset{N}{\sum }}}\ln \left[\frac{1}{\sqrt{2\pi {\sigma }^2}}\exp \left(-\frac{y_k-w{x}_k}{2{\sigma }^2}\right)\right]\end{array}$ (8)

Let $\beta =\frac{1}{{\sigma }^2}$

$\ln L\left(y/X\mathrm{,}w\mathrm{,}{\sigma }^2\right)=\frac{N}{2}\ln \left(2\pi \right)-\frac{N}{2}\ln \left(\beta \right)-\frac{\beta }{2}{\displaystyle \underset{k=1}{\sum }}N\left[{\left(y-f\left(x_k,w\right)\right)}^2\right]$

$-\ln L\left(y/X\mathrm{,}w\mathrm{,}{\sigma }^2\right)=\frac{\beta }{2}{\displaystyle \underset{k=1}{\sum }}N\left[{\left(y-f\left(x_k,w\right)\right)}^2\right]-\frac{N}{2}\ln \left(2\pi \right)+\frac{N}{2}\ln \left(\beta \right)$ (9)

Since $\ln \left(2\pi \right)$ and $\ln \left(\beta \right)$ are constant, formula (9) can be written as:

$-\ln L\left(y/X\mathrm{,}w\mathrm{,}{\sigma }^2\right)=\frac{\beta }{2}{\displaystyle \underset{k=1}{\overset{N}{\sum }}}\left[{\left(y-f\left(x_k,w\right)\right)}^2\right]+\text{constant}$ (10)

Now, maximizing (-y) is equivalent to minimizing (y). This being the case, formula (10) corresponds exactly to the minimization of deviations obtained by the algebraic approach to formula (4). So the least squares method is the result of the frequentist approach to solving the problem of estimating the parameters of a linear model under the assumption of a normal noise distribution [12] . But maximum likelihood estimation of the variance of noise sigma² is often underestimated or overestimated [13] . If the number of parameters and data evolve simultaneously, or if the density function is not convex, this phenomenon is at the root of overlearning or underlearning, a key property of this approach.

Having determined the optimal value of the parameters w and $\beta$, we can now make a prediction for a new value of x. The probability distribution of y knowing $x\mathrm{,}w\mathrm{,}\beta$ can then be formulated as follows:

$P\left(t\mathrm{<mo>\; |\; </mo>}x\mathrm{,}{w}^{\mathrm{<mo>\; *\; </mo>}}\mathrm{,}{\beta }^{\mathrm{<mo>\; *\; </mo>}}\right)=N\left(t\mathrm{<mo>\; |\; </mo>}y\left(x\mathrm{,}{w}^{\mathrm{<mo>\; *\; </mo>}}\right)\mathrm{,}{\beta }^{-1}\right)$ (11)

This approach provides good results when we have a large quantity of data and the relationship between the variables x and y are linear. But when the sample size is very small, it can lead to overlearning and poor prediction results [1] . That’s why another approach is possible in such circumstances.

In practice, we’ll be using python’s scikit-learn package for least-squares parameter estimation.

The least-squares approach produces good results when the properties contained in the data exist, including linearity, independence of observations, constant variance of the error term and normality of the latter. However, this method can be ineffective if the data sample contains outliers, multicollinearity between variables and small data size. The latter characteristic results in overfitting.

We can also take a Bayesian approach to parameter estimation. In this case, it is important to define an a priori probability for the parameters and an a posteriori probability obtained by Bayes’ theorem. The maximum a posteriori method will be used to incorporate the a priori distribution into parameter estimation.

The a priori probability distribution on the parameters is the probability on the parameters before having observed the data. It will be denoted by

$P\left(w\mathrm{<mo>\; |\; </mo>}\text{alpha}\right)=N\left(\mathrm{0,}{\alpha }^{-1}I\right)$.

The a posteriori distribution is an update of the beliefs or parameter distribution after taking into account the a priori distribution and the observed data. Using Bayes’ theorem, we can express it as follows:

$P\left(w\mathrm{<mo>\; |\; </mo>}X\mathrm{,}\alpha \mathrm{,}{\beta }^{-1}\right)=\frac{L\left(Y/X\mathrm{,}w\mathrm{,}{\beta }^{-1}\right)\cdot P\left(w\mathrm{<mo>\; |\; </mo>}\alpha \right)}{P\left(Y/X\right)}$

The a posteriori distribution is directly proportional to the a priori distribution and likelihood.

$P\left(w\mathrm{<mo>\; |\; </mo>}X\mathrm{,}\alpha \mathrm{,}{\beta }^{-1}\right)\propto L\left(Y/X\mathrm{,}w\mathrm{,}{\beta }^{-1}\right)\cdot P\left(w\mathrm{<mo>\; |\; </mo>}\alpha \right)$

We can thus determine w by looking for the most probable value of w given the observed data, i.e. by maximum a posteriori (MAP). By introducing the logarithm into formula (11), we obtain:

$w^{*}=\ln \left[L\left(Y|X,w,{\beta }^{-1}\right)\right]P\left(w|\alpha \right)$

$w^{*}=\ln \left[L\left(Y|X,w,{\beta }^{-1}\right)\right]N\left(0,{\alpha }^{-1}I\right)$

$w^{*}=\ln \left[L\left(Y|X,w,{\beta }^{-1}\right)\right]+\ln \left(N\left(0,{\alpha }^{-1}I\right)\right)$

$w^{*}=-\frac{\beta }{2}{\displaystyle \underset{k=1}{\overset{N}{\sum }}}{\left(y-g\left(x_k,w\right)\right)}^2+\frac{\alpha }{2}{w}^{\text{T}}w$

If we refer to Equation (1), Formula (13) becomes:

$-\frac{\beta }{2}{\displaystyle \underset{k=1}{\overset{N}{\sum }}}{\left(y-w^{\text{T}}\varphi \left(x_k\right)\right)}^2+\frac{\alpha }{2}{w}^{\text{T}}w$

With $\varphi \left(x\right)$ a basis function that transforms the initial sample data so that a linear model can be used in the problem formulation. We also note that maximizing the posterior probability is equivalent to minimizing the sum of squared regularized errors.

This has the effect of reducing the phenomenon of overfitting observed in the frequentist approach. Probabilities thus prove to be a powerful tool for estimating the parameters of a linear model.

Formula (15) is used to make a point estimate of the w parameter values. It corresponds to the Ridge regression of L₂. When the error distribution follows a Laplace distribution in formula 14, we obtain the Ridge regression of L₁ [3] . But if we want to make an estimate in terms of an interval, we’ll use Bayes’ approach in all its completeness, taking into account the multiplication and sum rule in probability.

This involves representing (15) as a probability distribution. This distribution is the predictive distribution. Consider X and $y$ the training data. Given a new point x, the task is to predict the value y. We need to formulate and calculate the expression $P\left(y\mathrm{<mo>\; |\; </mo>}x\mathrm{,}X\mathrm{,}y\right)$. The parameters $\alpha$ and $\beta$ are known in advance, as they can be calculated from the data.

The predictive probability distribution is given by applying Bayes’ theorem using the multiplication and sum rule in probability theory:

$P\left(y\mathrm{<mo>\; |\; </mo>}x\mathrm{,}X\mathrm{,}y\right)={\displaystyle \int P\left(y\mathrm{<mo>\; |\; </mo>}x\mathrm{,}w\right)P\left(w\mathrm{<mo>\; |\; </mo>}X\mathrm{,}y\right)\text{d}w}$ (12)

This expression can be calculated analytically by:

$P\left(y\mathrm{<mo>\; |\; </mo>}x\mathrm{,}X\mathrm{,}y\right)=N\left(y\mathrm{<mo>\; |\; </mo>}m\left(x\right)\mathrm{,}{s}^2\left(x\right)\right)$ (13)

The probability distribution $P\left(y\mathrm{<mo>\; |\; </mo>}x\mathrm{,}X\mathrm{,}y\right)$ is equivalent to $P\left(w\mathrm{<mo>\; |\; </mo>}y\right)$.

$m\left(x\right)=\beta \varphi {\left(x\right)}^{\text{T}}S{\displaystyle \underset{n=1}{\overset{N}{\sum }}}\varphi \left(x_n\right)y_n$ (14)

$s^2\left(x\right)\mathrm{<mo>\; =\; </mo>}{\beta }^{-1}+\varphi {\left(x\right)}^{\text{T}}S\varphi \left(x\right)$ (15)

With the matrix S equal to:

$S^{-1}=\alpha I+\beta {\displaystyle \underset{n=1}{\overset{N}{\sum }}}\varphi \left(x_n\right)\varphi {\left(x_n\right)}^{\text{T}}$ (16)

Applying Bayes’ approach to estimating the parameters of a linear model yields the same results as minimizing the variance of the regularized errors. But the Bayes approach has the advantage of using only the training sample [1] . Given a small sample size and prior knowledge of the parameter distribution, it gives excellent results. Minimization by the regularized error function requires another validation sample for hyper-parameter optimization [14] .

In practice, the Monte Carlo Chain Markov algorithm and the variational inference algorithm, which we won’t be presenting here, are available for estimating the probability a posteriori. These two algorithms can be used to approximate the integral of formula (16). In Python, they are contained in the pycm3 and Bayepy packages respectively.

The following assumptions must be met in order to use the Bayes approach:

The results of the Bayesian approach can be sensitive to the choice of a priori distributions. Inappropriate a priori specifications can lead to biased estimates. This approach is also influenced by outliers if the a priori specifications are not well chosen.

Table 1 presents the advantages and disadvantages of the two approaches, after having elucidated the characteristics of each of them.

We’re going to estimate the parameters of a linear model using least squares on the one hand, and the Bayesian approach on the other. To do this, we have chosen to use the Python language. The sklearn package will be used to implement least squares, while the pymc3 and Bayespy packages will be used to implement the Bayesian approach.

Let’s consider formula (21) and estimate the parameters. Here are the results:

Let’s now consider the function $\sin x+\epsilon$ with $\epsilon ~\mathcal{N}\left(\mathrm{0,0.3}\right)$ having generated the data sample. This function can be estimated using a polynomial function of degree 3. This degree is obtained after a model selection step such as cross-validation, etc., which we won’t go into here. We will now evaluate the different methods on the function $a_0+a_{1}x+a_2{x}^2+a_3{x}^3$.

So the Bayesian approach doesn’t seem to be very effective compared with least squares when faced with reduced data, since only the generated data doesn’t have a complex structure. By contraposition, this may therefore confirm the hypothesis that the Bayesian approach is efficient in the face of reduced data that have a complex structure within them.