Using the likelihood function to find an approximate confidence region for several parameters (often only one, in which case we speak of a confidence interval) of a specific distribution goes back to the classic publication of Kendall and Stuart [1] . To understand the rest of our article, it is necessary to briefly review their main result, summarized by

Theorem 1. Assuming a case of regular (meaning the distribution’s support is not a function of any of the distribution’s parameters) estimation, the following random variable

2 ln L ( X ; θ ^ ) − 2 ln L ( X ; θ 0 ) (1)

has approximately the chi-square distribution with K degrees of freedom, where K is the number of parameters to be estimated, X is the set of n observations (allowing for the possibility of a multivariate distribution), L ( X ; θ ) denotes the corresponding likelihood function, θ ^ is the vector of the resulting ML estimates, and θ 0 represents the true (but unknown) values of the parameters.

Proof. Realizing that

ln L ( X ; θ ) : = ∑ i = 1 n ln f ( x i ; θ ) (2)

where x i are the individual observations, then differentiating the RHS with respect to θ and setting each component of the answer to 0 yields

∑ i = 1 n ∂ ln f ( x i ; θ ) ∂ θ = 0 (3)

where ∂ ∂ θ implies differentiating ln f ( x i ; θ ) with respect to each parameter; the result is thus a vector with K components.

To solve the corresponding set of equations, we expand the LHS of (3) at θ = θ 0 , thus getting

∑ i = 1 n ∂ ln f ( x i ; θ ) ∂ θ | θ = θ 0 + ∑ i = 1 n ∂ 2 ln f ( x i ; θ ) ∂ θ 2 | θ = θ 0 ( θ − θ 0 ) + ⋯ = 0 (4)

where the second derivative of ln f ( x i ; θ ) is a symmetric K by K matrix. Dividing each side of the last equation by n, we solve it for θ − θ 0 getting, to the first-order accuracy

θ ^ − θ 0 ≃ − μ 2 − 1 Y (5)

where

Y : = 1 n ∑ i = 1 n ∂ ln f ( x i ; θ ) ∂ θ | θ = θ 0 (6)

μ 2 : = E ( ∂ 2 ln f ( X ; θ ) ∂ θ 2 | θ = θ 0 ) (7)

Note that the expected value of Y is 0 , since

∫ All x ∂ ln f ( x ; θ ) ∂ θ f ( x ; θ ) d x = ∫ All x ∂ f ( x ; θ ) ∂ θ d x = ∂ ∂ θ ( 1 ) = 0 (8)

and its variance-covariance matrix equals to − μ 2 / n , due to

d d θ ∫ All x ∂ ln f ( x ; θ ) ∂ θ f ( x ; θ ) d x = μ 2 + μ 11 = O (9)

where

μ 11 : = E ( ∂ ln f ( X ; θ ) ∂ θ ∘ ∂ ln f ( X ; θ ) ∂ θ | θ = θ 0 ) (10)

is the variance-covariance matrix of ∂ ln f ( X ; θ ) ∂ θ | θ = θ 0 (the small circle implies direct product of the two vectors), and O is the zero matrix.

Similarly expanding (1) and utilizing (5) we get, to the same level of accuracy

2 n Y ( θ ^ − θ 0 ) + n ( θ ^ − θ 0 ) T μ 2 ( θ ^ − θ 0 ) = − n Y T μ 2 − 1 Y (11)

resulting in the following simple approximation to (1)

− n Y T μ 2 − 1 Y n (12)

where Y n has (due to Central Limit Theorem) a K-variate Normal distribution with the mean of 0 and the variance-covariance matrix of − μ 2 . Introducing Z = Y n , the moment generating function of (12) is then computed by

∫ − ∞ ∞ exp ( − z T ( − μ 2 ) − 1 z 2 + t   z T ( − μ 2 ) − 1 z ) d z ( 2 π ) K / 2 det ( − μ 2 ) = det ( − μ 2 1 − 2 t ( − μ 2 ) − 1 ) = ( 1 − 2 t ) − K / 2 (13)

easily identified as the MGF of χ K 2 distribution. ■

The result then leads to a simple (we call it basic) algorithm of constructing the corresponding confidence region based on a random independent sample of n observations.

From the last theorem it follows that an approximate confidence region is found by following these steps:

● Using sample values drawn from the distribution, maximize its likelihood function, thus getting the ML estimates of all parameters;

● Choose a confidence level (denoted τ ) and make (1) (where θ ^ represents the ML estimates, but θ 0 are now variables to be solved for) equal to the corresponding critical value of the χ K 2 distribution;

● Solve for θ 0 , resulting in two limits of the corresponding confidence interval when K = 1 , a closed curve when K = 2 , a closed surface when K = 3 , etc.; this means that, beyond a one-parameter case, the resulting region can be displayed only graphically (a trivial task for today’s computers).

The following Mathematica program shows how it is done, first using a Geometric distribution with a (computer generated) sample of 40 and confidence level of 90%, then a Gamma distribution with a sample of 60 and confidence level of 95%.

The last line of the program has produced the following 95% confidence region (see Figure 1) for α (horizontal scale) and β (vertical scale).

A three-parameter estimation can be done in a similar manner, only the call to “ContourPlot” needs to be replaced by “ContourPlot3D”. Visualizing a confidence region for more than three parameters would require slicing it into a sequence of cross-sections.

The error of this procedure depends on the sample size, decreasing with n in the O ( 1 n ) manner; it can reach several percent for small samples. The purpose of the next section is to show how to substantially reduce this error, making it rather insignificant in all practical situations.

There have been many attempts at improving the accuracy of this approximation, starting with Bartlett [2] , followed by [3] and many others, all concentrating on higher moments of the approximately Normal distribution of ML estimators. Our approach is different: we aim at finding an appropriate correction to the χ 2 distribution of the Likelihood function. The technique we use is not unlike that of [4] , in spite of diverging objectives.

It has been shown in previous publications [5] and [6] that the distribution of (1) is more accurately described by the following probability density function

χ K 2 ( u ) ( 1 + A n ⋅ ( u K − 1 ) ) (14)

where χ K 2 ( u ) is the chi-square PDF of the original theorem, A is a quantity (often a simple constant) specific to the sampled distribution, and K is the number of parameters to be estimated. In terms of the corresponding CDF, this becomes

1 − Γ ( K 2 , u 2 ) + 2 A n K ( u 2 ) K / 2 exp ( − u 2 ) Γ ( K 2 ) (15)

(the two-argument Γ denotes the incomplete Gamma function) or, more explicitly

e r f ( u 2 ) − A n 2 u π exp ( − u 2 )

when K = 1 , and

1 − ( 1 + A u 2 n ) exp ( − u 2 ) (16)

when K = 2 (the two most important cases).

Note that the expected value of distribution (14) is K + 2 A n ; similarly expanding (1) then yields the formula for computing A, given the PDF of the sampled distribution.

Leaving out less important details, we now indicate the individual steps of such derivation.

● Extend the LHS of (4) by two more terms of the corresponding expansion, namely by

+ 1 2 ∑ i = 1 n ∂ 3 ln f ( x i ; θ ) ∂ θ 3 | θ = θ 0 ( θ − θ 0 ) 2 + 1 6 ∑ i = 1 n ∂ 4 ln f ( x i ; θ ) ∂ θ 4 | θ = θ 0 ( θ − θ 0 ) 3 + ⋯ (17)

Note that the third and fourth derivatives of ln f ( x i ; θ ) constitute fully symmetric tensors of ranks 3 and 4 respectively; each is further multiplied by the corresponding power (one less than the derivative’s order) of θ − θ 0 . The multiplication is carried out by taking the dot product along one of the tensor’s indices with vector θ − θ 0 , and repeating as many times as the power of θ − θ 0 indicates; the result is thus always a vector with one remaining index.

● Solve the resulting equation (iteratively) for θ ^ − θ 0 , to the same order of accuracy. This extends our existing solution (5), which we denote Θ 1 , by the following quadratic term

Θ 2 = − μ 2 − 1 ( Y 2 − μ 2 + 1 2 μ 3 Θ 1 ) Θ 1 (18)

and a cubic term, given by

Θ 3 = − μ 2 − 1 ( 1 2 ( Y 3 − μ 3 ) Θ 1 2 + μ 3 Θ 2 Θ 1 + 1 6 μ 4 Θ 1 3 + ( Y 2 − μ 2 ) Θ 2 ) (19)

where

Y l : = 1 n ∑ i = 1 n ∂ l ln f ( x i ; θ ) ∂ θ l | θ = θ 0 (20)

μ l : = E ( ∂ l ln f ( X ; θ ) ∂ θ l | θ = θ 0 ) (21)

(we have already explained how to multiply a symmetric tensor by a power or a product of vectors).

● Similarly expand (1); further divided by the sample size n, this results in the following scalar expression

Y Θ ( 1 ) + Y Θ 2 − 1 6 μ 3 Θ 1 3 + Y Θ 3 − 1 2 μ 3 Θ 2 Θ 1 2 − 1 6 ( Y 3 − μ 3 ) Θ 1 3 − 1 12 μ 4 Θ 1 4 + ⋯ = Y Θ ( 1 ) + ( Y 2 − μ 2 ) Θ 1 2 + 1 3 μ 3 Θ 1 3 + 1 3 ( Y 3 − μ 3 ) Θ 1 3 + 1 4 ( μ 3 Θ 1 2 ) × ( μ 3 Θ 1 2 )       + 1 12 μ 4 Θ 1 4 + ( μ 3 Θ 1 2 + ( Y 2 − μ 2 ) Θ 1 ) × ( ( Y 2 − μ 2 ) Θ 1 ) + ⋯ (22)

where the × symbol between two vectors (not used beyond this formula) implies an operation we call contraction of the two vectors, carried out by

V 1 × V 2 : = − V 1 μ 2 − 1 V 2 (23)

This is also how we perform a contraction of two specific indices of a tensor (required shortly); e.g. contracting the first two indices of μ 3 would be done (and denoted) by

μ 3 ˙ : = ∑ i , j = 1 K ( μ 3 ) i , j , n ( − μ 2 − 1 ) i , j (24)

resulting in a vector.

● Finally, take the expected value of (22) to get K n + 2 A n 2 + ⋯ .

This yields (term by term)

K n + μ 211 + K n 2 + μ 3 μ 111 3 n 2 + μ 31 n 2 + 2 μ 3 μ 3 + μ 3 ˙ μ 3 ˙ 4 n 2 + μ 4 4 n 2 + 2 μ 3 μ 21 + μ 3 ˙ μ 21 + μ 21 ∘ ∘ μ 21 + μ 21 ∘ μ 21 ∘ + μ 22 − K n 2 (25)

where

μ 21 : = E ( ∂ 2 ln f ( X ; θ ) ∂ θ 2 ∘ ∂ ln f ( X ; θ ) ∂ θ | θ = θ 0 ) (26)

with μ 211 , μ 111 , μ 22 and μ 31 defined analogously. Bold-face notation indicates that each term of (25) has been contracted in all of its indices, resulting in a scalar. For some of these terms, there are several possibilities to carry out such a contraction (e.g. the six indices of μ 21 μ 21 can be contracted in altogether five non-equivalent ways, only two of which we actually need); it is thus very important to indicate (in the ambiguous cases only; note that all pair-wise contractions of μ 4 , and also of μ 31 , yield the same answer) the proper way of doing it.

A dot implies contraction of two of the indicated indices ( μ 3 ˙ thus making the result into a vector, while μ 211 implies, by the absence of a dot over 2, that the first two indices have been contracted with the third and fourth index, rather than with each other); similarly, two circles indicate that the corresponding two indices are to be contracted: μ 21 ∘ ∘ μ 21 thus makes the first index of μ 21 ∘ ∘ contract with its last index, returning a vector yet to be contracted with the first index of μ 21 (due to the lack of a dot over 2); similarly μ 21 ∘ μ 21 ∘ indicates contracting the last index of μ 21 ∘ with the first index of μ 21 ∘ , while the remaining two indices of μ 21 ∘ are contracted with the remaining two indices of μ 21 ∘ .

Since μ 3 μ 111 = − μ 3 μ 3 − 2 μ 3 μ 21 , which can be verified in a manner of (9), the final formula for 2A then reads

μ 211 + μ 31 + μ 22 + 1 4 μ 4 + 1 4 μ 3 ˙ μ 3 ˙ + 1 6 μ 3 μ 3 + μ 3 ˙ μ 21 + μ 3 μ 21 + μ 21 ∘ ∘ μ 21 + μ 21 ∘ μ 21 ∘ (27)

when K = 1 , the formula simplifies to

A = μ 211 + μ 31 + μ 22 + 1 4 μ 4 2 ( − μ 2 ) 2 + 5 12 μ 3 2 + 2 μ 3 μ 21 + 2 μ 21 2 2 ( − μ 2 ) 3 (28)

We should mention that, in principle, the 1 n -proportional correction in (14) has two more terms of increasing complexity (see [5] ), namely

χ K 2 ( u ) ( 1 + A n ⋅ ( u K − 1 ) + B n ⋅ ( u 2 K ( K + 2 ) − 2 u K + 1 ) + C n ⋅ ( u 3 K ( K + 2 ) ( K + 4 ) − 3 u 2 K ( K + 2 ) + 3 u K − 1 ) ) (29)

where the B and C coefficients can be found in a manner very similar to how we have just established the value of A; most surprisingly, both B and C then turn out to be equal to zero.