The Beta-Binomial model accommodates overdispersion in binary data by allowing the heterogeneity in the success probability.

The moments of the Beta-binomial distribution are characterized by the following expressions:

The marginal mean is:

Najera-Zuloaga et al. [10] assumed that the probability parameter p i of the beta-binomial is linked to the covariates through the logistic transformation:

By the relationship in Equation (2), we define the linear predictor as:

This explicit separation of the mean, link, and linear predictor ensures compatibility with the general GLM framework while allowing over-dispersion through the Beta mixing distribution [5][6][10].

The variance function can be written as:

Substituting p i = μ i m in (4) we have:

where ρ = ϕ 1 + ϕ is the intra-class correlation parameter and an alternative parameterization of the over-dispersion parameter. The variance exceeds the Binomial variance μ i m ( m − μ i ) by the multiplicative inflation factor 1 + ( m − 1 ) ρ that explicitly models over-dispersion. When ρ = 0 , the classical binomial model is recovered.

For the higher order moments, we will make the same substitutions p i = μ i m and ρ = ϕ 1 + ϕ .

The third central moment is defined as:

The fourth central moment is defined as:

The Negative Binomial is constructed in the same manner. Counts are Poisson conditional on a Gamma-distributed rate. Marginalising over the rate yields a distribution whose moments are stated below in the heterogeneity parameterisation of Lindén and Mäntyniemi [3].

The moments of the Negative Binomial distribution are as follows.

The mean function is specified using a log-link:

so that

The linear predictor is therefore expressed as:

This log-link ensures that the predicted counts remain positive for all covariate values and provides a natural interpretation of the regression coefficients as multiplicative effects on the mean count [11].

The variance function Var ( Y i ) = r ( 1 − p i ) p i 2 can be written with proper substitutions for p i = r μ i + r and r = 1 α as:

The variance exceeds the mean by the quadratic term α μ i 2 with α quantifying the degree of overdispersion. When α = 0 , the model reduces to the Poisson distribution [12][13]. This Poisson-Gamma mixture construction and its variance structure are standard in the literature [11]-[13].

The higher-order moments for the Negative-binomial ( μ i , α ) distribution are obtained through appropriate substitutions for p i = r μ i + r and r = 1 α .

The third central moment is defined as:

The fourth central moment is defined as:

The two models share this construction but differ in the features that matter for penalized estimation. Both attach a single scalar to the excess variation, both nest a one-parameter model at the boundary of the dispersion space, and both supply the four moments required in Section 3. They differ in support, in link, and in how the variance responds to a change in the mean. Table 1 sets these features side by side. The rows on which the models differ determine the results of Section 4.

Table 1. Structural contrast between the two over-dispersed models.

The third and fourth central moments are derived in this section because the Estimating Functions (EFs 3.0) of Section 3 require them. The first two moments fix the mean and the variance and the orthogonalization step draws on skewness and kurtosis to recover information those two leave behind.

Let y i , i = 1 , ⋯ , n be independent observations from unspecified distribution with mean equation E ( y i ) = μ i .

Where, μ i = μ i ( β ) , and variance equation be defined as:

where ϕ is the dispersion parameter.

A candidate set of unbiased Estimating Functions (EFs) is:

It is easy to show both are unbiased, E ( π j i ) = 0 , for j = 1 , 2 .

Utilizing Theorem 2.2 of Godambe and Thompson [8], elementary orthogonal EFs in (16) for i = 1 , 2 , 3 , ⋯ , n are constructed using the Gram-Schmidt orthogonalization procedure based on the set of unbiased EFs in (15). This orthogonalization removes redundancy between moment conditions and minimizes asymptotic variance.

where, π 2 i * is the orthogonalized π 2 i and κ 3 i is the skewness coefficient defined in (17).

Using the elementary EFs in (16), Theorem 2.2 of Godambe and Thompson [8] is applied to construct a linear combination of elementary EFs in (18)

The optimal coefficients follow the same projection in every case. For elementary estimating function h i and parameter θ j ,

The regression equation in (20) and the dispersion equation in (21) are coupled through the fitted mean since the dispersion parameter is identified from residuals taken about the fitted mean rather than the true mean. The penalty is applied to (20) alone. This coupling accounts for most of the results in Section 4.

Among all unbiased EFs constructed from the moment conditions of the over-dispersed models, the OEFs attain the maximum Godambe information [7][8]. This is equivalent to minimizing the asymptotic variance of the joint regression-dispersion estimator θ = ( β , ϕ ) .

Let S ( θ ) denote the sensitivity matrix which is the expected derivative of the estimating function with respect to the parameter vector.

Let V ( θ ) denote the variability matrix which is the variance of the estimating function.

The asymptotic covariance of the estimator obtained by solving the joint OEF system is the Godambe sandwich covariance in (22).

The dispersion block of (22) supplies the standard error of the dispersion estimate.

In this study, Johnson et al. [14] general penalized estimating functions framework for the parameter vector β = ( β 1 , ⋯ , β p ) T based on a random sample of size n is adopted to construct penalized OEFs.

The penalized version of (20) is given as:

where, q λ = p ′ λ is the derivative a penalty function p λ ( β ) .

Four penalty functions are considered.

Let β = ( β 1 , ⋯ , β p ) T denote the coefficient vector in regression setting and let λ > 0 be a tuning (regularization) parameter.

The Ridge penalty ( ℓ 2 -penalty) by Hoerl & Kennard [15] imposes quadratic shrinkage on all coefficients.

The Lasso penalty ( ℓ 1 -penalty) by Tibshirani [16] induces sparsity through an absolute-value penalty in (25).

The ℓ 1 -penalty performs simultaneous feature shrinkage and selection.

The Adaptive Lasso [17] introduces coefficient-specific weights and can achieve oracle properties under regularity conditions.

where, β ˜ j is an initial consistent estimator such as the unpenalized OEF utilized in this study and γ > 0 controls the degree of adaptivity. The Adaptive Lasso is a non-convex penalty that provides stronger shrinkage for weak signals and reduced bias for large coefficients.

The Smoothly Clipped Absolute Deviation (SCAD) penalty [18] reduces bias for large coefficients while retaining sparsity.

The SCAD penalty is obtained by integration:

The penalty is defined primarily by its first derivative p ′ ( β ) :

where, a > 2 is the shape parameter (commonly a = 3.7 ) Fan and Li [18] and ( x ) + = max ( x , 0 ) .

Penalized OEFs do not derive from a likelihood and so deviance-based criteria and the classical information criteria are unavailable [6][19]. The tuning parameter is instead chosen by K -fold cross-validation on the estimating functions themselves. For each candidate value we solve the training folds while penalizing the regression equation and leaving the dispersion equation unpenalized. Further, we score the resulting estimate by the squared norm of the unpenalized joint OEF evaluated on the held-out fold. Moment-based sample splitting of this kind is established in the method-of-moments literature [20][21].

The cross-validated OEF loss is defined as:

where, K is the number of folds, g β , k * and g ϕ , k * denote the jointly optimal estimating functions for regression and dispersion parameters, respectively, as presented in (20) and (21) and evaluated using only observations in fold k of size n k . The function q λ ( ⋅ ) is the derivative of a penalty function p λ ( β ) with respect to β , sgn ( β ) is the component-wise sign function and ‖ ⋅ ‖ is the Euclidean norm.

The third and fourth order quantities entering (29) are analytic functions of the model parameters. They come from the moment expressions in (6) and (7) for the Beta-Binomial and from (12) and (13) for the Negative Binomial and are evaluated at the estimate obtained from the training folds. No moment of any order is estimated from the held-out observations. The held-out fold contributes an average of as many analytic terms as it holds observations. The size of the fold therefore governs the Monte Carlo noise of that average and not the estimation of a fourth order quantity. The criterion then averages over the K folds and every observation enters exactly once. The effective sample behind the criterion is rather than n divided by K . Small folds therefore leave the criterion intact. However, the variability matrix in (22) is built differently. It substitutes residuals about the fitted mean into an empirical average and Section 4.1.4 reports how that construction behaves at n = 100 .

Cross-validation curves for penalized estimators are often flat near the minimum under over-dispersion, so the raw minimiser gives unstable models [22]. The tuning parameter is therefore taken from a k -standard-error ( k -SE) band around the minimum:

Throughout this paper we use k = 1 . This gives the standard 1-standard-error (1-SE) rule popularized in recent literature [23][24]. The band runs from the smallest λ whose loss lies within one standard error of the minimum to the largest. So, its right edge is the sparsest model not distinguishable from the minimum. The choice within the band is method specific. For LASSO and Adaptive LASSO the right edge is taken. For Ridge the geometric mean of the two edges is used since Ridge shrinks without selecting and over-shrinkage at the right edge inflates the dispersion estimate. The candidate values lie on a logarithmic grid and so this point is the arithmetic midpoint of the band on the scale in which the search is conducted. SCAD is tuned by the same band augmented with the stability criterion of Meinshausen and Bühlmann [25]. The concave penalty admits several local solutions and the loss surface alone does not separate them. In this case, the sparsest λ is retained whose selection frequency stays within 5% of the best value.

This section presents results of the finite-sample dispersion inference of the penalized OEFs in the Beta-Binomial (BB) Logistic Regression model. The BB model accommodates overdispersion in grouped binary data through an intra-cluster correlation parameter ρ that controls excess variability beyond the binomial case [10]. The variance function is as given in Equation (7) which exceeds the binomial variance by the multiplicative inflation factor 1 + ( m − 1 ) ρ .

The data-generating process (DGP) is such that, for each replication, p = 20 covariates are drawn from a mean-zero multivariate normal distribution with a first-order autoregressive correlation structure (AR(1)) with correlation 0.5. This is a moderate level that introduces collinearity without extreme multicollinearity. Given a coefficient vector β , the success probability for i t h observation follows the logistic link in (4) and the expected count is as in (3) with m = 40 trials per cluster. The response is then drawn as Y i ~ Beta-Binomial ( m , μ i , ρ ) generated through the Beta-Binomial mixture in which the binomial success probability is itself Beta-distributed with shape parameters a = μ i ( 1 − ρ ) / ( m ρ ) and b = ( m − μ i ) ( 1 − ρ ) / ( m ρ ) , such that ρ is exactly the intra-cluster correlation. The covariates are standardized within each replication and R = 500 independent datasets are generated per scenario under a common random seed to ensure that all six estimators are compared on identical data.

4.1.1. Simulation Design

Four scenarios are constructed by crossing two overdispersion levels ρ = 0.20 (moderate over-dispersion) and ρ = 0.65 (heavy over-dispersion) with two active-set sizes s = 4 and s = 8 while holding sample size n = 100 , number of features p = 20 , cluster size m = 40 , and R = 500 Monte Carlo replications. Resultantly, the sparsity ratios are s / p = 0.2 (sparse) and s / p = 0.4 (moderately dense). The shift from s / p = 0.2 to s / p = 0.4 is included to probe how the penalized OEF framework behaves under stress rather than to evaluate a regime for which the methods were optimally designed. Table 3 summarizes the design.

Table 3. Design of the four Beta-Binomial simulation scenarios.

n = sample size; p = candidate predictors; s = active predictors; s / p = sparsity ratio; ρ true = intra-cluster correlation.

Under this design, six estimators are evaluated: the Quasibinomial GLM, the unpenalized OEF, Ridge (L2-OEF), L1_plain (LASSO-OEF), Adaptive LASSO, and SCAD. The Quasibinomial GLM is the standard moment-based benchmark that estimates β by iteratively reweighted least squares with a quasi-binomial variance and ρ by the Pearson Method-of-Moments (MoM) χ 2 statistic. The unpenalized OEF is the Godambe efficient benchmark that jointly estimates the regression and dispersion parameters through the set of orthogonalized estimating functions in (20) and (21) which incorporate the third and fourth-order moment information. The four penalized estimators apply L1, adaptive L1, L2, and SCAD penalties to the regression equation.

4.1.2. Hyperparameter Selection and Tuning Diagnostics

Hyper-parameter selection for the BB model follows the cross-validated OEF loss of Equation (29) with the k -SE band and method-specific selection rules as discussed in Section 3.3. For each candidate λ , the regularization parameter is chosen by K − fold cross-validation ( K = 5 ) of the held-out OEF norm. The remainder of this sub-section reports the BB-specific behavior of this criterion.

We apply the k-SE selection rule and its standard-error band presented in Equation (29). For L1_plain and Adaptive LASSO, λ * is taken as the right k -SE value which represents the largest λ whose cross-validated loss remains within the 1-SE band. This selection favours the sparsest model that the data do not distinguish from the cross-validation minimum.

The choice of λ within the band differs for Ridge. Over-shrinking β worsens the mean fit and the dispersion equation compensates by absorbing the unexplained variability into ρ . The dispersion estimate is inflated as a result. The same coupling is reported for over-dispersed models generally where variability that the mean model leaves unexplained is taken up by the dispersion parameter [4][26][27]. The right k -SE value deepens the over-shrinkage and the left edge under-regularizes. Therefore, we set λ * to be the geometric mean of the left and right k -SE values, which is a midpoint rule on the scale of the search. Candidate values of λ are generated on a logarithmic grid as is standard for penalized regression paths [22][28]. The geometric mean of the two edges is the arithmetic midpoint of the band on that grid and the selected value satisfies log λ = ( log λ L + log λ R ) / 2 with λ L and λ R as the two edges. The arithmetic mean of the same two edges falls close to the right edge in log units and sits near the most heavily shrunk model in the band. The cross-validated loss stays within one standard error of its minimum everywhere in the band by construction [29]. The midpoint therefore reduces dispersion bias at a bounded cost in predictive adequacy and needs no further tuning constant.

Finally, SCAD requires yet another adjustment as the 1-SE rule alone does not suffice for it. This is because the concave penalty produces characteristically flat CV surfaces in the intermediate λ region and because the penalty is nonconcave the optimization objective may admit several local solutions [18][28][30] leaving the loss insensitive to λ across a wide range.

Stability selection, as introduced by Meinshausen and Bühlmann [25] gauges whether a variable belongs in the model by how reliably it is chosen when the data or tuning parameters are perturbed. We therefore tune SCAD with a stability-augmented rule ( τ = 0.90 ; guard =1.05×) such that λ * is placed where the CV loss first turns sharply upward and reverting to the CV minimum whenever the safeguard is not met. The threshold τ is the smallest selection frequency at which a coefficient is treated as reliably retained and is conventionally fixed within the 0.6 - 0.9 range [25]. On the other hand, the guard borrows the logic of the 1-SE rule by admitting the stable λ only when its loss stays within 5% of the best value so that predictive accuracy is not sacrificed [31].

4.1.3. Dispersion Coverage

Consider first the two sparse scenarios s / p = 0.2 . Table 4 and Table 5 report unconditional 95% Wald coverage for the dispersion parameter. For each case, two dispersion Confidence Interval (CI) constructions are compared: a direct Wald interval on the ρ scale and a delta-method interval via logit ( ρ ) ( θ -Wald). The direct interval is given by:

where z 1 − α / 2 = 1.96 for 95% coverage and SE ^ ( ρ ^ ) taken as the square root of the corresponding diagonal entry of the Godambe (sandwich) covariance in Equation (22). The θ -Wald construction however maps the interval from the logit ( ρ ) scale on which the sampling distribution may be more symmetric, back to the ρ -scale. The θ -Wald interval is obtained by the delta method [32].

Writing θ = logit ( ρ ) , the standard error is transformed as:

and the interval is formed on the θ scale and mapped back through the inverse logit such that:

which confines the interval for ρ to (0, 1) and improves coverage when the sampling distribution of ρ ^ is asymmetric [33].

Dispersion coverage varies sharply across methods. Under direct ρ -Wald, L1_plain and Adaptive LASSO come close to nominal (0.928 and 0.924) whereas the unpenalized OEF (0.556), Ridge (0.546), and SCAD (0.582) all under cover. The θ -Wald transformation leaves the L1-type methods essentially unchanged (0.928 and 0.916) and lifts the others only slightly (unpenalized OEF 0.616, Ridge 0.598, SCAD 0.588). The limited gain is expected: a transformation of this kind improves coverage mainly when the untransformed sampling distribution is skewed and adds little when it is already close to symmetric [34][35]. Since ρ is confined to (0, 1), the distribution of ρ ^ is fairly symmetric to begin with so mapping through the logit results to little correction.

Table 4. Dispersion coverage: Scenario 1 ( s = 4 , ρ = 0.20 ).

Next, we consider Scenario 2 ( ρ = 0.65 ) which is a harder setting for inference. Ridge’s coverage drops to 0.520 implying that roughly half of its dispersion intervals miss the true ρ . The θ -Wald transformation has little effect in this simulation Scenario, 0.732 vs. 0.718 for the unpenalized OEF.

Table 5. Dispersion coverage: Scenario 2 ( s = 4 , ρ = 0.65 ).

Increasing the active set to s = 8 moves the design into a moderately dense regime ( s / p = 0.4 ). Table 6 and Table 7 report the results.

Table 6. Dispersion coverage: Scenario 3 ( s = 8 , ρ = 0.20 ).

Scenario 4 ( ρ = 0.65 ) amplifies these tendencies further. Ridge’s ρ coverage drops to 0.328, the lowest in all scenarios. The explanation to this is the shape of ρ ^ under Ridge at high over-dispersion. Its distribution is multimodal and much of its mass is centred away from ρ true to cover it.

Table 7. Dispersion coverage: Scenario 3 ( s = 8 , ρ = 0.65 ).

4.1.4. The Quasi-Binomial Comparison

Tables 4-7 contain a comparison worth treating on its own. The Quasi-Binomial GLM estimates dispersion by the Pearson Method-of-Moments (MoM) statistic and uses no third or fourth moment information. It attains dispersion coverage of 0.948, 0.942, 0.912 and 0.874 across the four scenarios. The unpenalized OEF is the Godambe-efficient member of the class and the benchmark for the penalized variants. It attains 0.556, 0.732, 0.530 and 0.740. The estimator with the smaller asymptotic variance produces the less accurate intervals in every scenario. The penalized variants do not share this. They attain between 0.924 and 0.964 and exceed the quasi-binomial in the two heavy over-dispersion scenarios. The failure is therefore specific to the unpenalized sandwich interval.

The two columns measure the same thing. The quasi-binomial interval is constructed for the dispersion multiplier 1 + ( m − 1 ) ρ and mapped to the ρ scale by a monotone linear inversion which preserves coverage exactly. The coverage of ρ obtained in this way equals the coverage of ϕ in every scenario.

Godambe optimality concerns the asymptotic variance of the estimator. It states that no other unbiased estimating function in the class attains a smaller variance in the limit [8][36]. A Wald interval requires two further conditions, that is, that the estimated standard error is close to the true standard error and that the normal approximation holds at the sample size in use. Optimality addresses neither [37].

The difficulty lies in the dispersion block. The variability matrix entering the sandwich covariance in (22) requires the variance of the dispersion estimating function. That function is constructed from squared deviations and so its variance depends on fourth-order moments which is why the construction in Section 3.1 carries the skewness and kurtosis coefficients explicitly. Here those moments are estimated from residuals taken about a fitted mean with 21 parameters fitted from 100 observations and so a large share of the variation is absorbed by the fit. The estimated variability matrix is too small and the standard error inherits the deficiency. An interval of nominal width 1.96 standard errors is then too narrow to attain its stated coverage.

The Pearson estimator does not incur this cost. Its interval is referred to a chi-square distribution on n − p degrees of freedom rather than constructed from an estimated variance. The degrees of freedom adjustment accounts for the fitted parameters directly and no fourth-order moment is estimated anywhere. The interval takes its shape from a reference distribution rather than estimating that shape from fewer than five observations per fitted parameter. The efficiency the OEF gains asymptotically is returned in the finite-sample estimation of its own variance and at n = 100 the second effect is the larger.

The logit-scale interval was introduced to correct skewness in the sampling distribution, and it does not improve coverage. The gain is at most 6 percentage points across the four scenarios and in half the cells of Tables 4-7 the transformed interval covers less often than the direct one. A transformation moves a misplaced interval and does not widen a narrow one. The deficiency is therefore one of width.

The quasi-likelihood advantage is confined to this model. Under the Negative Binomial the ordering reverses. Quasi-Poisson dispersion coverage falls from 0.832 to 0.596 as over-dispersion and signal density increase, while the unpenalized OEF holds between 0.580 and 0.907 once the log-scale interval is used. The advantage under the Beta-Binomial comes from the bounded support and not from the moment estimator.

All four penalized variants are solved from the same estimating system as the unpenalized OEF. They use the same dispersion equation in (21) and take their standard errors from the same dispersion block of (22). They differ only in the penalty term applied to the regression equation. L1_plain covers between 0.904 and 0.954 across the four scenarios and Adaptive LASSO between 0.912 and 0.964, against 0.530 to 0.740 for the unpenalized fit. Coverage near nominal under the same construction points to the estimated variability matrix as the source of the undercoverage rather than to the estimating system. Ridge covers between 0.328 and 0.546 and SCAD between 0.582 and 0.822. Section 4.1.3 traces the Ridge figures to the multimodal shape of the dispersion estimate under over-shrinkage and Section 4.1.2 traces the SCAD figures to local solutions of the nonconcave objective. Both failures follow from the penalty and neither is a property of the interval construction.

The undercoverage has a known source. Residuals taken about a mean fitted with p parameters have an expected sum of squares reduced by approximately the factor 1 − p / n [6]. With 21 parameters at n = 100 that reduction is substantial and the effect is larger for a fourth order quantity. The estimated variability is therefore too small and the standard error built from it is too small and so an interval of nominal width is narrower than its stated coverage requires. Narrow intervals explain part of the undercoverage but not all of it. The rest is consistent with the dispersion estimating function departing from normality at n = 100 which makes a Wald interval the wrong shape as well as the wrong width. Both problems ease as the sample increases. Mancl and DeRouen [38] address the same problem by inflating residuals to account for the leverage of the fitted mean.

Downward bias of an empirical sandwich variance in small samples is documented in the estimating equation literature [39][40]. Wald intervals built from it undercover for the same reason. The corrections proposed there address the regression block of the sandwich under clustering. What appears here is the same effect acting on a dispersion block. That block draws its variability from fourth order quantities and the attenuation compounds with the order of the moment.

At n = 100 the Godambe interval for the dispersion parameter in the unpenalized Beta-Binomial fit is best read alongside a moment-based or resampling interval. The L1-type fits do not need that support and hold coverage above 0.900 in every scenario, and Section 4.3 explains why.

4.1.5. Dispersion Estimation Accuracy

Tables 8-11 report bias and RMSE for the dispersion parameter estimate ρ ^ . van de Geer et al. [41] note that reporting both quantities separates systematic distortion from total estimation error.

Table 8. Bias and RMSE: Scenario 1 ( s = 4 , ρ = 0.20 ).

Table 9. Bias and RMSE: Scenario 2 ( s = 4 , ρ = 0.65 ).

Table 10. Bias and RMSE: Scenario 3 ( s = 8 , ρ = 0.20 ).

Table 11. Bias and RMSE: Scenario 4 ( s = 8 , ρ = 0.65 ).

A central finding in this paper is that penalization disturbs dispersion inference substantially. Under the BB model, the RMSE ρ ^ varies by a factor of three across penalties, from 0.079 for the Adaptive LASSO to 0.250 for Ridge. Under the NB, the effect is larger still with bias ranging from −0.052 to −0.563 within Scenario 4 alone.

The mechanism behind this is the mean-dispersion coupling of the BB variance function that was described earlier in Section 4.1.2. Errors in β propagate through the success probability in Equation (4) into the dispersion equation where they are absorbed as shifts in ρ ^ . Therefore, penalizing β alone alters the residual structure used to estimate variability and the dispersion equation compensates. This is not a numerical anomaly but a structural feature of the joint OEF system, one that standard penalized GLM procedures do not expose because they treat dispersion as ancillary.

At ρ = 0.20 all penalized methods show modest ρ ^ bias but differ sharply in stability. SCAD carries only a small positive pooled bias of 0.019 in Scenario 1 and 0.024 in Scenario 3 yet the largest RMSE of all methods (0.153 and 0.164) nearly twice that of the unpenalized OEF. This mix of near-zero bias with inflated RMSE points to a heavy-tailed rather than systematically shifted ρ ^ distribution.

The histograms (Figure 1 and Figure 3) show SCAD’s cluster just below ρ true alongside a distinct secondary cluster near 0.65 implying the squared-error summary is driven by this right-tail minority. This is consistent with the multiple local solutions admitted by the nonconvex SCAD penalty [18][28][30], under which selection instability feeds into dispersion variability through the coupled system.

L1_plain, by contrast, achieves both a small bias of −0.024 in Scenario 1 and the tightest ρ ^ concentration among the penalized methods (RMSE ρ ^ = 0.085 ).

Ridge gives the largest RMSE ρ ^ at ρ = 0.20 of 0.125 in Scenario 1 and 0.209 in Scenario 3. This reflects a multimodal ρ ^ distribution. The multimodal structure arises from the over-shrinkage mechanism described in Section 4.1.2. When λ is large, Ridge over shrinks β thus making the mean fit to worsen. This results in the dispersion equation accommodating the residual variability by inflating ρ ^ and thus producing the right-tail hump visible in Figures 1-4. This is precisely why the geometric mean K -SE rule for Ridge we introduced in Section 4.1.2 is key as it selects the weakest penalization within the acceptable band. Resultantly, the penalty avoids the over shrinkage-dispersion compensation cycle.

At ρ = 0.65 all methods underestimate ρ , including SCAD which has a bias of −0.060 in Scenario 2 and −0.063 in Scenario 4. The unpenalized OEF is the most stable method with consistently low RMSE ρ ^ (0.087 - 0.094) and moderate negative bias (−0.024 to −0.074). This indicates that much of the dispersion instability under the penalized methods is penalty-induced rather than intrinsic to the OEF system. At this high ρ , L1_plain, Adaptive LASSO, and SCAD give comparably concentrated ρ ^ distributions, RMSE ρ ^ spans 0.079 - 0.107. This is a clear contrast to the low ρ regime in which SCAD is the most volatile. Under this over-dispersion regime, Ridge remains the most volatile overall recording a RMSE ρ ^ of 0.250 in Scenario 4.

Figures 1-4 display per-method histograms of ρ ^ for the four scenarios.

Figure 1. Per-method histograms of ρ ^ : Scenario 1 ( s = 4 , ρ = 0.20 ).

Figure 2. Per-method histograms of ρ ^ : Scenario 2 ( s = 4 , ρ = 0.65 ).

Figure 3. Per-method histograms of ρ ^ : Scenario 3 ( s = 8 , ρ = 0.20 ).

Figure 4. Per-method histograms of ρ ^ : Scenario 4 ( s = 8 , ρ = 0.65 ).

This section reports dispersion inference for the penalized OEFs under Negative-Binomial (NB2) regression, where the variance is quadratic in the mean. The NB2 model carries over-dispersion in a heterogeneity parameter α through the variance function in Equation (11) and nests the Poisson model in the limit α → 0 [11][12][42]. The conditional mean uses the log link of Equation (8) and so the fitted counts stay positive. The exponential mean bears directly on penalty behaviour. A coefficient acts multiplicatively on the mean under this link, so shrinkage in β produces multiplicative rather than additive distortion in the fitted mean μ i [11][42]. The quadratic variance function then carries that distortion into the dispersion estimate, as Sections 4.2.3 and 4.2.4 show.

The data-generating process follows this specification. For each replication p = 20 covariates are drawn from a mean-zero multivariate normal distribution with first-order autoregressive correlation AR(1) and correlation 0.5. Given a coefficient vector β the conditional mean for the i t h observation follows the log link of Equation (8) and the mean count of Equation (9). The response is drawn as Negative-Binomial Y i ~ NB ( μ i , α ) with the variance of Equation (11). As in the Beta-Binomial study of Section 4.1, R = 500 independent datasets are generated per scenario under a common random seed. The sample size is raised to n = 200 here to accommodate the heavier tails of count responses.

4.2.1. Simulation Design

Four scenarios cross two over-dispersion levels α = 0.25 and α = 0.75 with two active set sizes s = 4 and s = 8 holding n = 200 , p = 20 and R = 500 replications fixed. The VMR is 1 + α μ i . Table 12 gives the design.

Table 12. Design of the four negative-binomial simulation scenarios.

n = sample size; p = candidate predictors; s = active predictors; s / p = sparsity ratio; α true = NB2 heterogeneity.

The same six estimators are evaluated as in Section 4.1: the Quasi-Poisson GLM, the unpenalized OEF, Ridge (L2-OEF), L1_plain (LASSO-OEF), Adaptive LASSO and SCAD. The Quasi-Poisson GLM estimates β by iteratively reweighted least squares under a Quasi-Poisson variance and estimates dispersion by the Pearson Method of Moments statistic. The unpenalized OEF estimates the regression and dispersion parameters jointly through the orthogonalized EFs in (20) and (21). The penalized estimators apply their penalties to the regression equation only.

4.2.2. Hyperparameter Selection and Tuning Diagnostics

Tuning follows the cross-validated OEF loss in (29) and the method-specific k -SE rules of Section 3.3, the same framework used for the BB model. For L1_plain and Adaptive LASSO, λ * is the right k -SE value within the 1-SE band. For Ridge, it is the left edge which avoids the over-shrinkage compensation described in Section 4.1.2 and which operates here with an amplified effect under the exponential mean. SCAD uses the stability-augmented rule of Section 4.1.2 with the NB2 settings τ = 0.80 and guard = 1.10×. The guard falls back to the CV minimum in all four scenarios.

4.2.3. Dispersion Coverage

Consider first the two sparse scenarios s / p = 0.20 . Table 13 and Table 14 report unconditional 95% Wald coverage for the dispersion parameter. Two interval forms are compared, as in Section 4.1.3. The first is a direct Wald interval on the α -scale. The second is a delta-method interval on the log ( α ) scale, denoted the η -Wald, which parallels the θ -Wald construction of Equations (31) to (33) with the log link in place of the logit.

Coverage of α under the direct Wald interval falls below nominal for the unpenalized OEF at 0.526 and for SCAD at 0.434. The η -Wald interval recovers most of this. The unpenalized OEF rises from 0.526 to 0.906 and Adaptive LASSO from 0.966 to 0.974. The transformation works because α is unbounded above, α ∈ ( 0 , ∞ ) , and the sampling distribution of α ^ is severely right skewed under every method. Consequently, the log scale symmetrizes it which is where a transformation of this kind gives most benefit [34][35]. Section 4.2.4 takes this up.

Table 13. Dispersion coverage: Scenario 1 ( s = 4 , α = 0.25 ).

Scenario 2, α = 0.75 repeats the pattern. The η -Wald gain is large. L1_plain rises from 0.352 to 0.882 and Adaptive LASSO from 0.388 to 0.940.

Table 14. Dispersion coverage: Scenario 2 ( s = 4 , α = 0.75 ).

Table 15 and Table 16 give the denser scenarios s / p = 0.4 . The move from s / p = 0.2 to s / p = 0.4 brings out a distinct NB2 pattern.

Table 15. Dispersion coverage: Scenario 3 ( s = 8 , α = 0.25 ).

Quasi-Poisson dispersion coverage falls from 0.832 in Scenario 1 to 0.596 in Scenario 4. The Pearson Method of Moments estimator deteriorates under the combined stress of high over-dispersion and dense signals.

Table 16. Dispersion coverage: Scenario 4 ( s = 8 , α = 0.75 ).

4.2.4. Dispersion Estimation Accuracy

Tables 17-20 report bias and RMSE for the heterogeneity parameter α .

Table 17. Bias and RMSE: Scenario 1 ( s = 4 , α = 0.25 ).

Table 18. Bias and RMSE: Scenario 2 ( s = 4 , α = 0.75 ).

Table 19. Bias and RMSE: Scenario 3 ( s = 8 , α = 0.25 ).

Table 20. Bias and RMSE: Scenario 3 ( s = 8 , α = 0.75 ).

As in the BB model, penalization disturbs α estimation far more than β estimation and the exponential mean structure amplifies the effect. The variance in Equation (11) grows with the square of the mean in Equation (9) and so errors in β propagate quadratically into the variance function and small β distortions produce large shifts in α ^ . At α = 0.25 every method except Ridge underestimates α and at α = 0.75 every method including Ridge underestimates α . The downward bias is most severe for Adaptive LASSO at −0.164 in Scenario 1 and −0.589 in Scenario 2, which matches its sharply concentrated α ^ distributions with mean α ^ = 0.086 against α true = 0.25 . Ridge runs the other way at α = 0.25 with a positive bias of 0.226 in Scenario 1 and 0.250 in Scenario 3 and the largest RMSE α ^ of any method at 0.624 and 0.467. At α = 0.75 its bias turns slightly negative and it becomes the most accurate in central tendency even though its RMSE α ^ stays high. The unpenalized OEF gives the most balanced accuracy of RMSE α ^ which places the instability of the L1-type penalties in the penalty rather than in the estimating system.

These behaviours follow from the geometry of the log link. The right skewness of α ^ under every method arises because the variance function couples α multiplicatively with μ 2 = exp ( 2 x T β ) . The exponential growth of μ 2 creates heavy right tails and asymmetric estimating equations for α , which is why direct Wald coverage is poor. This is a documented feature of interval estimation for the negative-binomial dispersion parameter [2][43], and it is why the η -Wald transformation on η = log ( α ) re-symmetrizes the dispersion geometry so effectively [44]. The same link accounts for the amplified Ridge bias. A coefficient acts multiplicatively on the mean under the log link [11][42], so linear shrinkage in β becomes multiplicative shrinkage in μ , most severely where α is high.

The multiplicative coupling through the exponential link is the structural reason dispersion inference is so penalty-sensitive under NB2.

Figures 5-8 display per-method histograms of α ^ for the four scenarios.

The η -Wald interval is the practical device for NB2 dispersion inference. Across the four scenarios the η -Wald procedure improves coverage well above direct Wald with the largest gains among the penalized methods and smaller gains for the unpenalized OEF. L1_plain η -Wald coverage exceeds 0.84 everywhere, ranging from 0.840 to 0.998 against direct Wald as low as 0.352. Adaptive LASSO improves from 0.348 to 0.958. SCAD is the exception. Its direct Wald coverage is already low at 0.316 to 0.468 and the η -Wald remains modest at 0.342 to 0.660. The cause is SCAD’s severe underestimation of α . A transformation re-symmetrizes an interval but cannot recenter it on a value the point estimate has missed and so with α ^ far below α true , the true value stays outside the interval and coverage barely moves.

The two simulation studies examined penalized OEF dispersion inference under Beta-Binomial logistic regression in Section 4.1 and Negative-Binomial regression in Section 4.2. They agree on the behaviour of the framework and disagree on almost every ranking within it. The disagreements trace to the structural differences between the models, that is, the bounded response and logit link of the BB model against the unbounded counts and log link of NB2.

Figure 5. Per-method histograms of α ^ : Scenario 1 ( s = 4 , α = 0.25 ).

Figure 6. Per-method histograms of α ^ : Scenario 2 ( s = 4 , α = 0.75 ).

Figure 7. Per-method histograms of α ^ : Scenario 3 ( s = 8 , α = 0.25 ).

Figure 8. Per-method histograms of α ^ : Scenario 4 ( s = 8 , α = 0.75 ).

Penalization affects dispersion inference more strongly than coefficient inference and this holds in both models. The mean-dispersion coupling of the joint OEF system in Equations (20) and (21) is the reason. Errors in β propagate through the mean function into the dispersion equation which carries no penalty and so shifts the dispersion estimate to restore the moment match. Penalization adds a second source of variation to the dispersion estimate. The tuning parameter is selected from the data in every replication so the dispersion estimate is a random function of it and its variance separates into the variance at a fixed tuning value plus the variance contributed by selecting that value. The first term is the one the unpenalized fit also carries. The second has no counterpart there and it is what the choice of point within the standard-error band controls. NB2 amplifies the coupling because errors propagate multiplicatively through Equation (9) and quadratically through Equation (11). The unpenalized OEF gives the most consistent dispersion RMSE across scenarios with RMSE ρ ^ from 0.087 to 0.108 under BB model and RMSE α ^ from 0.127 to 0.275 under NB2 model. The instability under the penalized methods is therefore penalty-induced and not intrinsic to the estimating system. Under BB the most volatile estimator is Ridge with RMSE ρ ^ up to 0.250, which follows the over-shrinkage compensation cycle of Section 4.1.2. Under NB2, the largest bias comes from Adaptive LASSO at high over-dispersion. The bias reaches −0.589 in Scenario 2 and −0.563 in Scenario 4, where α true = 0.75 and the mean falls to 0.161 and 0.187 respectively. This follows from its weight dependent shrinkage as reported in Section 4.2.4. At the lower over-dispersion level, the largest bias is Ridge in the opposite direction, at +0.226 and +0.250 in the two scenarios.

Whether a variance-stabilizing transformation helps is likewise model-dependent and for the same structural reason. Under NB2, the η -Wald interval on the log scale raises α coverage by as much as 61 percentage points while the analogous θ -Wald interval on the logit scale gains at most 6 points for ρ coverage under BB model. The dispersion geometry explains the difference. The NB2 heterogeneity parameter α ∈ ( 0 , ∞ ) is severely right-skewed under the exponential mean and the log scale symmetrizes it. The BB intra-cluster correlation ρ ∈ ( 0 , 1 ) is confined to the unit interval and is already close to symmetric and gains little from the logit scale. Sections 4.1.3 and 4.2.3 give the cell-level results.

SCAD’s reliability turns on the smoothness of the estimating-function geometry which again separates the two models. Under the bounded BB support SCAD is the most variable method for ρ ^ with dispersion RMSE ρ ^ spanning 0.153 to 0.164 at ρ = 0.20 despite small pooled bias, and the instability worsens at ρ = 0.65 . Under the smoother NB2 geometry, it is numerically stable across all four scenarios with reduced sensitivity only at the dense setting s / p = 0.4 . Both patterns follow from the multiple local solutions admitted by the nonconvex penalty [18][28]. The bounded BB response range worsens that multiplicity and the monotone NB2 score function suppresses it.