In this paper, the estimators of the scale parameter of the exponential distribution obtained by applying four methods, using complete data, are critically examined and compared. These methods are the Maximum Likelihood Estimator (MLE), the Square-Error Loss Function (BSE), the Entropy Loss Function (BEN) and the Composite LINEX Loss Function (BCL). The performance of these four methods was compared based on three criteria: the Mean Square Error (MSE), the Akaike Information Criterion (AIC), and the Bayesian Information Criterion (BIC). Using Monte Carlo simulation based on relevant samples, the comparisons in this study suggest that the Bayesian method is better than the maximum likelihood estimator with respect to the estimation of the parameter that offers the smallest values of MSE, AIC, and BIC. Confidence intervals were then assessed to test the performance of the methods by comparing the 95% CI and average lengths (AL) for all estimation methods, showing that the Bayesian methods still offer the best performance in terms of generating the smallest ALs.
The exponential distribution is one of the most commonly used continuous distributions applied in life data analysis. In particular, it is commonly used for systems exhibiting some form of constant failure rate. For a continuous random variable X = ( X 1 , X 2 , ⋯ , X n ) , the exponential distribution density function, the pdf, is thus given by
f ( X , λ ) = λ e − X λ , X ≥ 0. (1)
where λ is the constant failure rate parameter. For more details on exponential distribution and its applications, [
Empirical Bayes estimators of exponential distribution parameters have been introduced by multiple authors, such as [
This paper aimed to study the estimation of exponential distribution across a variety of loss functions, applying three different criteria-based methods to compare the resulting estimators. These were mean square error (MSE), the Akaike Information Criterion (AIC), and the Bayesian Information Criterion (BIC).
The remainder of this paper is thus organized as follows. In Section 2, the mathematical derivations of the estimation methods are present, while the final model selection is explained in Section 3. The Monte Carlo simulation study results are then used in Section 4 to perform a comparison of the estimation methods, based on applying the appropriate criteria for the mean square error (MSE), Akaike Information Criterion (AIC), and Bayesian Information Criterion (BIC), allowing a conclusion based on the results findings to be offered in Section 5.
This section outlines the various estimates of the parameter of the exponential distribution, as given in (1).
The maximum likelihood estimator (MLE) is a technique used for estimating the parameters of a given distribution as discussed in [
L ( X , λ ) = λ n e − λ ∑ i = 1 n X i .
Taking the natural logarithm of both sides yields
ln L ( X , λ ) = n ln λ − λ ∑ i = 1 n X i
and the MLE estimator of λ can thus be obtained by solving the following equation:
∂ ∂ λ ln L ( X , λ ) = 0.
Hence,
n λ − ∑ i = 1 n X i = 0 ,
and the maximum likelihood estimator, λ ^ MLE , is then given by
λ ^ MLE = n ∑ i = 1 n X i . (2)
This section demonstrates the process of deriving Bayesian estimates of the scale parameter for the exponential distribution. Three different loss functions are used to achieve this, which are the squared error loss function, the entropy loss function, and the composite LINEX loss function.
The gamma ( α , β ) can be considered as a conjugate prior of λ with its density function written in the form
h ( λ , α , β ) = β α Γ ( α ) λ α − 1 e − β λ , λ , α , β > 0 ,
where α and β are the shape parameter and scale parameter, respectively.
The posterior density function of λ for the given random sample X = ( X 1 , X 2 , ⋯ , X n ) is thus obtained as
π ( λ , X ) = ∏ i = 1 n f ( λ , X ) h ( λ , α , β ) ∫ 0 ∞ ∏ i = 1 n f ( λ , X ) h ( λ , α , β ) d λ = λ n e − λ ∑ i = 1 n X i β α λ α − 1 e − β λ ∫ 0 ∞ λ n e − λ ∑ i = 1 n X i β α λ α − 1 e − β λ d λ = λ n + α − 1 e − ( ∑ i = 1 n X i + β ) λ ∫ 0 ∞ λ n + α − 1 e − ( ∑ i = 1 n X i + β ) λ d λ .
This in turn implies that the posterior distribution can be written as
π ( λ , X ) = λ n + α − 1 e − ( ∑ i = 1 n X i + β ) λ ( ∑ i = 1 n X i + β ) n + α Γ ( n + α ) ,
which, as can be plainly observed, is a gamma distribution with parameters ( n + α ) and ( ∑ i = 1 n X i + β ) .
The three different loss functions used to develop a Bayes estimate for the parameter λ are discussed below.
The SE as discussed by [
L ( λ ^ , λ ) = ( λ ^ − λ ) 2 .
The Bayesian estimator of λ under the squared error loss function is the mean of the posterior density function. The Bayes estimator of λ under the squared error loss function is then denoted as λ ^ SE which can be written as
E ( λ | X ) = ∫ 0 ∞ λ π ( λ , X ) d λ = ( ∑ i = 1 n X i + β ) n + α Γ ( n + α ) ∫ 0 ∞ λ n + α e − ( ∑ i = 1 n X i + β ) λ d λ = Γ ( n + α + 1 ) ( ∑ i = 1 n X i + β ) n + α + 1 ⋅ ( ∑ i = 1 n X i + β ) n + α Γ ( n + α ) = n + α ∑ i = 1 n X i + β .
Such that,
λ ^ SE = n + α ∑ i = 1 n X i + β , (3)
where E ( . ) indicates the posterior expectation.
The entropy loss function, as discussed by [
L ( λ ^ , λ ) = K ( λ ^ λ − ln ( λ ^ λ ) − 1 ) .
The Bayes estimator of λ based on the entropy loss function, denoted by λ ^ BEN , is then given as
λ ^ BEN = ( E ( λ − 1 ) ) − 1 .
From this, it is possible to derive E ( λ − 1 ) as follows:
E ( λ − 1 ) = ∫ 0 ∞ λ − 1 π ( λ , X ) d λ = ( ∑ i = 1 n X i + β ) n + α Γ ( n + α ) ∫ 0 ∞ λ ( n + α − 1 ) − 1 e − ( ∑ i = 1 n X i + β ) λ d λ = ( ∑ i = 1 n X i + β ) n + α Γ ( n + α ) Γ ( n + α − 1 ) ( ∑ i = 1 n X i + β ) n + α − 1 .
The Bayesian estimation of λ based on the entropy loss function is therefore written as
λ ^ BEN = n + α − 1 ∑ i = 1 n X i + β . (4)
The composite LINEX loss function is defined by [
L ( λ ^ , λ ) = e − C ( λ ^ , λ ) + e C ( λ ^ , λ ) − 2, C > 0.
The Bayes estimator of λ based on composite LINEX loss function is denoted as λ ^ BCL and can be expressed as
λ ^ BCL = 1 2 C ln ( E e ( C λ | X ) E e ( − C λ | X ) ) .
To find E ( e ( C λ | X ) ) ,
E ( e ( C λ | X ) ) = ∫ 0 ∞ e ( C λ , X ) π ( λ , X ) d λ = ∑ i = 1 n X i + β Γ ( n + α ) ∫ 0 ∞ e − ( ∑ i = 1 n X i + β − C ) λ n + α − 1 d λ = ∑ i = 1 n X i + β ( ∑ i = 1 n X i + β − C ) n + α .
Here, in a similar manner, E ( e ( − C λ | X ) ) where
E ( e ( − C λ | X ) ) = ∑ i = 1 n X i + β ( ∑ i = 1 n X i + β + C ) n + α .
The Bayesian estimation of λ based on the composite LINEX loss function can thus be expressed as
λ ^ BCL = n + α 2 C ln ( ∑ i = 1 n X i + β + C ∑ i = 1 n X i + β − C ) . (5)
To compare the efficiency of MLE, SE, BEN, and BCL, estimators the mean square error, Akaike information criterion, and Bayesian information criterion methods were used to test their accuracy.
The mean square error references the mean squared distance between observed and predicted values.
The MSE is thus calculated as
MSE = ∑ i = 1 n ( λ i − λ ^ i ) 2 n , (6)
where λ ^ is the estimator of the parameter λ on the ith run and n is the sample size.
Estimates values with the lowest rates of MSE are preferred, as this means that λ ^ is closer to the actual values of λ .
The Akaike information criterion (AIC) is defined by the equation
AIC = 2 K − 2 log L ( λ ^ ) , (7)
where K is the number of estimated parameters and L ( λ ^ ) is the maximum value of the likelihood estimate of the parameters.
Higher values for the likelihood function give a better fit, with the minimum AIC; however, the value of the AIC increases as more parameters are added to the first component.
In the case of small data set, n K < 40 , the second-order AIC, AICc can be used more effectively. The AICc takes the form
AIC c = − 2 log ( λ ^ ) + 2 K + 2 K + 1 n − K − 1 , (8)
where 2 K + 1 n − K − 1 is the bias-correction factor. As n increases, 2 K + 1 n − K − 1 tends to zero; at that point, the AICc gives results that more closely resemble the AIC.
The Bayesian (or Schwarz) information criterion is expressed as
BIC = − 2 log L ( λ ^ ) + K log ( n ) . (9)
Further information on AIC and BIC can be found in [
The method with the smallest values of MSE, AIC, and BIC can, however, be assumed to be the most efficient method to estimate the parameter to estimate the exponential distribution, offering estimated values of λ close to its true value. The best method of estimating the parameter can also be determined by calculating which offers the highest log-likelihood value.
This section discusses the use of a Monte Carlo simulation to compare the MSE, AIC, and BIC criteria, as defined in (6), (7), and (9) respectively, to estimate the parameter of the exponential distribution based on the classical methods of comparison using MLE and Bayes estimators under the loss functions, including SE, BEN, and BCL, which can be computed as shown in Section 2 in equations 2 to 5. Sample sizes n = 10, 30, 150, 300, and 1,000 were used to achieve this, with data generated from the exponential distribution for the scale parameter λ = 1 , with an arbitrary prior parameter (α, β, C) = (1, 2, 0.5). The number of replications used was 1,000,000 for each sample size.
The efficiencies of the estimation methods are compared in
| Estimators | ||||
|---|---|---|---|---|
| n = 10 | ||||
| MLE | BSE | BEN | BCL | |
| λ ^ | 1.033686 | 1.003013 | 0.9978051 | 1.160923 |
| MSE | 0.166474 | 0.07241902 | 0.07111619 | 0.072879 |
| L | -6.350954 | -4.9385 | -13.81305 | -10.41417 |
| AIC | 15.07691 | 12.25219 | 30.00111 | 23.20334 |
| BIC | 15.00449 | 12.17978 | 29.92869 | 23. 13093 |
| n = 30 | ||||
| λ ^ | 0.9816438 | 0.991284 | 0.99964 | 1.005751 |
| MSE | 0.039308 | 0.030747 | 0.029941 | 0.030747 |
| L | -13.1564 | -15.2890 | -24.64788 | -13.21051 |
| AIC | 28.42004 | 32.68516 | 51.40291 | 28.52816 |
| BIC | 29.71409 | 33.9792 | 52.6969 | 29.8222 |
| n = 150 | ||||
| λ ^ | 0.995469 | 0.998004 | 0.9965661 | 1.001133 |
| MSE | 0.0069195 | 0.006578301 | 0.006535639 | 0.0065790 |
| L | -79.08826 | -75.59041 | -105.6675 | -81.1737 |
| AIC | 160.1765 | 153.2011 | 213.3351 | 164.3474 |
| BIC | 163.1871 | 156.1915 | 216.3457 | 167.358 |
| n = 300 | ||||
| λ ^ | 0.9949605 | 1.013197 | 0.9936484 | 0.9942531 |
| MSE | 0.003398617 | 0.0032999 | 0.003304556 | 0.003321389 |
| L | -183.2983 | -182.7589 | -165.4214 | -156.1926 |
| AIC | 368.5965 | 367.5179 | 332.8529 | 314.3853 |
| BIC | 372.3003 | 317.2217 | 336.5466 | 318.0891 |
| n = 1000 | ||||
| λ ^ | 1.001877 | 1.000891 | 1.001926 | 0.9990892 |
| MSE | 0.00100733 | 0.000980377 | 0.000997429 | 0.000996724 |
| L | -567.925 | -604.1954 | -555.8084 | -541.6347 |
| AIC | 1137.85 | 1210.389 | 1113.617 | 1085.269 |
| BIC | 1142.758 | 1215.297 | 1118.524 | 1090.179 |
| (L, U) | ||||
|---|---|---|---|---|
| AL | ||||
| n | estimators | |||
| MLE | BSE | BEN | BCL | |
| 10 | (0.7903452, 1.277027) | (0.83655, 1.169469) | (0.8462149, 1.149395) | (0.993913, 1.327934) |
| 0.4866818 | 0.332919 | 0.3031801 | 0.334021 | |
| 30 | (0.911776, 1.051511) | (0.928542, 1.054027) | (0.93897, 1.060325) | (0.9430085, 1.068498) |
| 0.139735 | 0.125485 | 0.121355 | 0.1254895 | |
| 150 | (0.9822032, 1.008736) | (0.9850244, 1.010984) | (0.9836738, 1.009458) | (0.988153, 1.014113) |
| 0.0265328 | 0.0259596 | 0.0257842 | 0.02596 | |
| 300 | (0.9878749, 1.001046) | (0.9999318, 1.012933) | (0.9871547, 1.000142) | (0.9877433, 1.000763) |
| 0.013197 | 0.0130012 | 0.0129873 | 0.0130197 | |
| 1000 | (0.9999109, 1.003843) | (0.9989327, 1.002849) | (0.9999695, 1.003882) | (0.9971325, 1.001046) |
| 0.0039321 | 0.0039163 | 0.0039125 | 0.0039135 | |
and the values of MSE, AIC, and BIC with log-likelihoods for the selected sample size, also offers the MSE values obtained using all four estimators approached. For small sample sizes (n = 10, 30), the BEN method can thus be seen to perform better than other estimation methods, based on it offering the smallest value of MSE. With an increasing sample size, however, the BSE method provides the lowest MSE. The results further indicate that the lowest values of AIC and BIC were obtained by using BCL as n increases further, while the values of AIC and BIC are close to each other for all estimation methods. Overall, all methods offer good performance with respect to the estimation of the parameter, though the higher the log-likelihood value, the better the parameter estimation method, and the results also suggest that the BCL method gives a higher log-likelihood value than all other approaches. Comparing the results in this paper to that in [
In this study, four methods of estimating the parameter of the exponential distribution were compared. Estimating the exponential parameter using classical MLE was thus compared with the use of the Bayesian method assessed using three criteria, the MSE, the AIC, and the BIC. The results indicate that the Bayesian method performs better than the maximum likelihood estimator for the estimation of the parameter, based on it having the smallest values of MSE, AIC, and BIC, with narrow 95% CIs and the shortest ALs.
Sincere thanks to the members of JAMP for their professional performance.
The author declares no conflicts of interest regarding the publication of this paper.
Alomari, H.M. (2023) A Comparison of Four Methods of Estimating the Scale Parameter for the Exponential Distribution. Journal of Applied Mathematics and Physics, 11, 2838-2847. https://doi.org/10.4236/jamp.2023.1110186