Regularized Models for Fitting Zero-Inflated and Zero-Truncated Count Data: A Comparative Analysis

This study adopts the use of Ridge regression and LASSO to fit count Health data with the aim of reducing the variance.

2.1 The ridge regression

The ridge regression is mathematically expressed as:

$\sum_{i=1}^n\left(y_i-\beta_0-\sum_{j=1}^p \beta_j x_{i j}\right)^2+\lambda \sum_{j=1}^p \beta_j^2=R S S+\lambda \sum_{j=1}^p \beta_j^2$    (1)

The first part of Eq. (1) is the regression sum of square (RSS), that, is $\sum_{i=1}^n\left(y_i-\beta_0-\sum_{j=1}^p \beta_j x_{i j}\right)^2=R S S$.

where, λ is the tuning parameter; y_i is observation i on the dependent variable; x_ij is the observation i on the independent variable j; β₀ is the intercept term and β_j is the Regression coefficient for independent variable j. These definitions are the same for other equations in this paper, where λ≥0 in (1) is the tunning parameter. The ridge regression estimates are similar to that of least square estimates and the choice of λ is what determines that. If λ=0, the ridge regression gives the same estimates as least square would. As λ increases the coefficient shrinks towards zero, therefore, a large λ produces lower estimates than a lower λ if used to model the same data set. The term on the right in Eq. (1), $\lambda \sum_{j=1}^p \beta_j^2$ is the shrinkage penalty. So, if no penalty is introduced, the estimates become least square estimates. The shrinkage penalty applies only to other coefficients and not the intercept, β₀ [7].

2.2 The LASSO regression

The challenge with ridge regression such as driving the regression estimates to zero as λ→∞ is overcome with Least Absolute Shrinkage and Selection Operator (LASSO). The model for LASSO is formulated as follows:

$\sum_{i=1}^n\left(y_i-\beta_0-\sum_{j=1}^p \beta_j x_{i j}\right)^2+\lambda \sum_{j=1}^p\left|\beta_j\right|$    (2)

Eqs. (1) and (2) are similar but the only difference is that penalty for ridge regression is $\beta_j^2$ while the penalty for LASSO is |β_j|. The estimates obtained from LASSO are much easier to interpret since it removed unimportant independent variables from the model. In other words, the shrinkage penalty in LASSO shrinks the coefficients of the independent variables having no significant effect on the dependent variables to zero.

In another manner, one can show that the LASSO and ridge regression coefficient estimates solve the problems, for LASSO we have:

$\left.\begin{array}{c}&\underset{\beta}{\operatorname{minimize}}\left\{\sum_{i=1}^n\left(y_i-\beta_0-\sum_{j=1}^p \beta_j x_{i j}\right)^2\right\} \\ &\text { subject to } \\ & \qquad \sum_{j=1}^p\left|\beta_j\right| \leq s\end{array}\right\}$  (3)

and for ridge,

$\left.\begin{array}{c}\underset{\beta}{\operatorname{minimize}}\left\{\sum_{i=1}^n\left(y_i-\beta_0-\sum_{j=1}^p \beta_j x_{i j}\right)^2\right\} \\ \text { subject to } \\ \qquad \lambda \sum_{j=1}^p \beta_j^2 \leq s\end{array}\right\}$    (4)

There is a corresponding value of s for every value of λ, such that the LASSO coefficient estimates for Eqs. (2) and (3) will be the same. The same can be said of Eqs. (1) and (4) that they will produce the same ridge regression coefficient estimates. Eq. (3) shows that LASSO coefficients have the smallest Residual sum of square in  $\left|\beta_1\right|+\left|\beta_2\right|<s$, while Eq. (4) shows that ridge coefficients have the smallest Residual sum of square in $\beta_1^2+\beta_2^2 \leq s$.

The ridge regression is likened to estimating β₁, β₂, …, β_p such that:

$\left\{\sum_{i=1}^p\left(y_i-\beta_0\right)^2+\lambda \sum_{j=1}^p \beta_j^2\right\}$     (5)

is minimized, and for LASSO, we have:

$\left\{\sum_{i=1}^p\left(y_i-\beta_0\right)^2+\lambda \sum_{j=1}^p\left|\beta_j\right|\right\}$     (6)

is minimized. It shows that, the ridge regression estimates can be obtained from Eq. (7) as follows:

$\hat{\beta}_j^R=y_j /(1+\lambda)$     (7)

and the LASSO estimates take the form:

$\hat{\beta}_j^L= \begin{cases}y_j-\lambda / 2, & \text { if } y_j<\lambda / 2 \\ y_j+\lambda / 2, & \text { if } y_j>\lambda / 2 \\ 0, & \text { if }\left|y_j\right| \leq \lambda / 2 .\end{cases}$     (8)

Software in the study [31] was used to analyze the data; the R package such as glmnet [32], dplyr [33], and tidyverse [34], Dwreg [35] were used for implementation.

2.3 Simulation study

The section contains the procedure for generated random variables that were used to fit the model. The response variable was simulated from discrete Weibull, and covariates from uniform distribution. To simulate zero inflated count data from discrete Weibull, the condition is that for over-dispersed, 0≤β≤1, and β≥2 for underdsipersed, for any chosen value of q. One thousand random response numbers were generated and five covariates were generated uniformly. The simulated data was fitted to four frequentist models, three Bayesian models, and the two regularized models considered in this study (Ridge and LASSO). For the choice of prior for Bayesian DW, Laplace prior was adopted using Metropolis-Hastings algorithm with an independent Gaussian proposal to draw samples from the posterior. Thirty thousand (30,000) iterations were performed. Specification of values for the parameters for the simulation experiment is given as follows: 10-fold Cross-validation was used to select the best value for the tuning parameter, λ and the Mean square error (MSE).

For overdispersed, we have:

$\left.\begin{array}{l}\theta_0=0.35, \theta_1=0.25, \theta_2=0.6, ? \\ \theta_3=0.6, \theta_4=0.7, \theta_5=0.66, \beta=0.8\end{array}\right\}$

For underdispersed, we have:

$\left.\begin{array}{l}\theta_0=0.35, \theta_1=0.25, \theta_2=0.6, ? \\ \theta_3=0.6, \theta_4=0.7, \theta_5=0.66, \beta=2.8\end{array}\right\}$

The equation of the relationship between the response and the covariates using log link is given in Eq. (9):

$\left.\begin{array}{rl}\log (q(1-q))= & 0.35+0.25 x_1+0.6 x_2 \\ & +0.6 x_3+0.7 x_4+0.66 x_5\end{array}\right\}$    (9)

The intervals for the uniformly generated random variables x ₁, x ₂, x ₃, x₄, and x₅ are (0, 1), (0, 2), (0, 1.5), (0, 3) and (0,1.8).

The mean and variance of simulated over-dispersed data is 73.311 and 20901.27 respectively. While the mean and variance of simulated under-dispersed data is 2.183 and 1.801.

Table 1. Model selection criteria for frequentist, Bayesian and regularized models over-dispersed

Table 1 is the results of model fit for simulated zero inflated count data show that DPMglmm outperformed other models based on minimum AIC and BIC values in the over-dispersed data. On the other hand, Ridge regression outperformed other models based on minimum AIC values, but not with BIC values in the under-dispersed data. For the under-dispersed simulated data, the cross-validation mean square error (MSE) for ridge regression is 1.828, while the cross-validation MSE for LASSO regression is 1.829. For the over-dispersed simulated data, the cross-validation MSE for ridge regression is 26797 while the MSE for LASSO regression is 26800.

In this investigation, health count data was subjected to modeling by employing a variety of frequentist, Bayesian, and regularized regression models, specifically, ridge and LASSO. Prior to the modeling of the life data, a simulation study encompassing frequentist models such as Poisson, Negative Binomial, Discrete Weibull, and Hurdle Negative Binomial was conducted. Additionally, the simulated data were modeled utilizing Bayesian models including DPMglmm, MCMCglmm, and Bayesian Discrete Weibull.

The findings from the simulation experiment revealed that, in the context of under-dispersed data, ridge regression surpassed LASSO and other frequentist and Bayesian models in terms of the Akaike Information Criterion (AIC). Conversely, based on the Bayesian Information Criterion (BIC), the Dirichlet Prior mixture model demonstrated superior performance. In terms of the Mean Squared Error (MSE), comparable performance between ridge and LASSO was observed, a pattern that manifested in the life data as well.

In conclusion, the outcomes of this study suggest that the superiority of ridge or LASSO in fitting count data is not absolute; instead, it is contingent upon the specific objectives of the study. If model selection is the primary goal, the use of the AIC suggests a preference for ridge regression, although their performance in terms of efficiency, as measured by the mean square error, is largely indistinguishable. Nevertheless, LASSO presents the unique characteristic of shrinking irrelevant variables to zero, thereby emphasizing the significant ones.