The Rectified Linear Unit activation function (ReLU) [1]-[11], along with its modifications such as leaky ReLU [12]-[15], is widely used in deep learning (DL), especially in two-dimensional (2D) and three-dimensional (3D) image analysis. We focus on input layers in 2D or 3D image analysis, but our arguments easily extend to intermediate layers and multi-dimensional problems. ReLU is given by h ( a ) = max ( 0 , a ) and can be expressed as y = h ( x ′ w + u ) when random noise is present. For image analysis, xand w represent vectors of pixel values and their corresponding weights, respectively. Analyzing data with ReLU is similar to censored regression or Tobit models [16] (pp. 973-977) that are widely studied in statistics and econometrics [17]-[19]. Tobit was introduced by Tobin [20] in his analysis of household expenditures on durable goods. However, knowledge from these studies has not been fully incorporated into DL research. The results derived from econometrics and statistics resemble those of DL, and incorporating these applications may enhance the performance of DL methods.

Pooling techniques [21]-[26], especially max pooling [27]-[32], have also been widely used. However, max pooling suffers from a serious drawback: when noise is large and signal is small, it often produces incorrect results. Song et al. [33] consider median-pooled gradients and note that “median pooling is suitable to reduce the effect of noise” [33] (p. 140). Nevertheless, their study lacks a solid theoretical framework, and median pooling itself is rarely employed.

Nawata [34] proposes the Grouping Adjusted Median Estimator (GAME), which is robust to non-normality and heteroscedasticity in the Tobit model. Grouping is conceptually similar to pooling, and the asymptotic distribution of GAME has been derived. In this paper, we examine the application of GAME to the Tobit model and present results from Monte Carlo experiments. When a signal represents a small percentage of the area, we can use an appropriate percentile. The concept of the “median” can be generalized to arbitrary percentiles.

We consider 2D images where S is the rectangle defined by 0 < x 1 ≤ 1 and 0 < x 2 ≤ 1 . x 1 and x 2 are divided into 300 equidistant grid lines, yielding n = 90,000 points per image. Let x 1 κ denote the κ -th grid line in x 1 and x 2 ℓ denote the ℓ -th grid line in x 2 . The intersection of x 1 κ and x 2 ℓ is denoted as x κ ℓ . We consider the models

The boundary of the region is given by

The true parameter values of w 0 , w 1 , and w 2 are 0.0, 1.0, and −1.0, respectively.

We compare three estimators:

1) CTMLE (assuming normal error terms),

2) GAME considered in this paper, and

3) MPE (max pooling estimator), which uses the maximum values within groups.

In contrast to MPE, GAME is based on the asymptotic normality of median.

For the i.i.d. case, we consider two distributions:

Case 1: Standard normal distribution, and

Case 2: Quasi-Cauchy distribution.

Since the estimation procedure failed to converge in many trials under the standard Cauchy distribution, we use the quasi-Cauchy distribution, where the error terms are drawn from U (0.002, 0.998) as a fat-tailed distribution. The variance of the distribution is 101.7.

For the heteroscedastic distributions, we consider the following cases:

Case 3: Heteroscedastic distribution I, u κ ℓ = [ 1 ( x 1 κ < 0.5 ) + 2 ∗ 1 ( x 1 κ ≥ 0.5 ) ] ε , and

Case 4: Heteroscedastic distribution II, u κ ℓ = [ 2 ∗ 1 ( x 1 κ < 0.5 ) + 1 ( x 1 κ ≥ 0.5 ) ] ε .

1 ( ⋅ ) is an indicator function that takes the value 1 if the argument is true and 0 otherwise, and ε is a standard normal random variable. For GAME and MPE, the sample space of ( x 1 , x 2 ) , denoted by S, is divided such that each group contains nine intersection points determined by three neighboring grid lines of x 1 and x 2 . The number of groups is 100 × 100 = 10,000. We perform 1000 trials using EViews 13 for each case. CLMLE and GAME can be efficiently estimated using standard programs without incurring significant computational cost.

Tables 1-4 present the Monte Carlo results. When the error terms follow the i.i.d. normal distribution (Table 1, Case 1), the biases, standard deviations (SDs), and mean squared errors (MSEs) of CTMLE are small. This outcome is quite reasonable, since CTMLE is not only consistent but also efficient in this case. The biases of GAME are very small, but the SDs and MSEs are larger than those of CTMLE. For MPE, the biases, SDs, and MSE are small for w 1 and w 2 , but that for w 0 is large (1.485). Figure 2 illustrates the boundaries of the two regions calculated from the true parameter values and the averages of w 0 * and w 1 * values. The boundaries obtained from the true parameters, CTMLE, and GAME are quite similar and nearly coincide. In contrast, the boundary given by MPE is almost parallel to the other lines.

Table 1. Standard normal distribution: Case 1.

SD: Standard deviation, MSE: Mean squared error.

Figure 2. Boundaries obtained from the true parameter values (True), CTMLE, GAME, and PME for the standard normal distribution: Case 1 in Table 1.

When the error terms follow the quasi-Cauchy distribution (Table 2, Case 2), the biases of CTMLE are large, with values of 2.124, −0.818, and 0.096 for w 0 , w 1 , and w 2 , respectively. In contrast, the biases, SDs, and MSEs of GAME are much smaller than those of CTMLE, with biases of −0.032, 0.073, and −0.075, respectively. MPE performs quite poorly, exhibiting not only very large biases but also very large SDs. Figure 3 illustrates the boundaries obtained from the true parameter values and the averages of the estimated parameters. Both CTMLE and MPE produce incorrect boundaries; however, GAME yields an almost correct boundary much more accurate than that from CTMLE or MPE.

Table 2. Quasi-Cauchy distribution: Case 2.

Figure 3. Boundaries obtained from the true parameter values (True), CTMLE, GAME, and PME for the quasi-Cauchy distribution: Case 2 in Table 2.

Table 3 and Table 4 present the heteroscedastic distribution results. For Heteroscedastic distribution I (Table 3, Case 3), GAME clearly reduces biases for all parameters. The biases of MPE for w 0 , w 1 , and w 2 are 1.097, 2.248, and −0.019, respectively, which are large for w 0 and w 1 but relatively small for w 2 . Since error term variances depend only on x 1 , this may be reflected in the Monte Carlo results. Figure 4 illustrates the boundaries calculated from the true parameter values and the estimates. The GAME boundary is closer to the true line than that from CTMLE, whereas MPE yields a poor result.

For Heteroscedastic distribution II (Table 4, Case 4), the biases and MSEs of GAME are smaller than those of CTMLE for all parameters. For MPE, although the bias for w 2 is small (−0.0017), the biases for w 0 and w 1 are large, at 3.339 and −2.299, respectively. Figure 5 shows the boundary lines of the two regions. GAME clearly improves upon CTMLE. As before, MPE performs poorly. In particular, P [ y i > 0 ] decreases as x 1 increases, and the estimated target and background regions become opposite to the true ones in Case 4.

Table 3. Heteroscedastic distribution I: Case 3.

Figure 4. Boundaries obtained from the true parameter values (True), CTMLE, GAME, and PME for Heteroscedastic distribution I: Case 3 in Table 3.

Table 4. Heteroscedastic distribution II: Case 4.

Figure 5. Boundaries obtained from the true parameter values (True), CTMLE, GAME, and PME for Heteroscedastic distribution II: Case 4 in Table 4.

ReLU and max pooling are widely used in DL. However, reliable outcomes cannot be obtained if the distribution of error terms is misspecified or if inappropriate methods are applied. The Monte Carlo results support this claim. Both CTMLE and MPE exhibit substantial biases and large MSEs under non-normal (fat-tailed) or heteroscedastic distributions.

GAME is a semiparametric estimator and is consistent under very general assumptions. Although the SDs of GAME are slightly larger than those of CTMLE, GAME clearly outperforms CTMLE under non-normal (fat-tailed) and heteroscedastic distributions. When ε i follows an i.i.d. standard normal distribution, the expected value of ζ = max { 0 , ε 1 , ε 2 , ⋯ , ε 9 } is 1.485, which coincides with the bias of MPE for w 0 . The biases for w 1 and w 2 are very small, at 0.003 and -0.004, respectively. The estimated boundary is nearly parallel to the true one. Many studies using ReLU exclude the constant term. This may be related to the fact that the max pooling method yields useful results in standard cases. In the other situations, however, MPE is a poor estimator not only for the constant but also for the slope coefficients. MPE produces highly inaccurate boundaries. In particular, when the variance is a decreasing function of x 1 (Case 4), the boundary slope becomes negative, and the estimated region where P [ y i > 0 ] is opposite to the true one.

Figure 6 illustrates an example of max, average, and median pooling, where the maximum, average, and median are calculated from nine cells using ReLU ( η ) = max { 0 , η } , with η following the standard Cauchy distribution. In this figure, no signal (S) exists; all non-zero values represent random noise (N), and the signal-to-noise ratio (S/N) is zero. Max pooling yields a very high value of 79, as if important information were contained in those cells. Average pooling [42]-[45] also produces relatively large values (9.33). In contrast, median pooling results in zero. Max pooling emphasizes large values in the observations. While this may be useful for detecting very weak signals in the sample space, it can lead to incorrect results when noise is substantial and S/N is low. As high- or ultra-high-resolution 2D or 3D images with considerable noise (small S/N) are becoming increasingly common, careful treatment to eliminate or reduce noise is essential. In DL application areas, such as medical or satellite imaging where high noise is common and robustness is most critical, the approach considered in this paper may be quite useful.

Figure 6. Example of max, average, and median pooling under the Cauchy distribution.

The ReLU and max pooling methods are widely used in DL. The Tobit model, commonly applied in econometrics and statistics, is closely related to ReLU. However, the knowledge obtained from the Tobit model has not been fully incorporated into DL. In this paper, we considered ReLU estimation using the Tobit framework combined with a grouping approach. The CTMLE, which assumes i.i.d. normal errors, exhibits large biases under fat-tailed (quasi-Cauchy) or heteroscedastic error distributions. We considered GAME, which combines grouping, adjustment, the median, and a weighted Tobit method. GAME is robust, remaining consistent under both non-normal and heteroscedastic error distributions. Moreover, unlike in binary cases, its asymptotic distribution is obtainable.

Monte Carlo results show that GAME outperforms CTMLE under non-normal or heteroscedastic errors. In the i.i.d. normal case, where CTMLE is efficient, GAME’s loss of efficiency is small; its biases and SDs remain close to those of CTMLE. In contrast, the MPE performs poorly, exhibiting large biases, SDs, and MSEs. In some cases, the region MPE suggests for P [ y i > 0 ] is even the opposite of the true region. Max pooling selects the maximum value in each group, which can help detect weak signals but also amplifies noise when the signal-to-noise ratio is low. Thus, max pooling requires caution, and noise-reducing methods such as GAME are essential.

The author would also like to thank an anonymous reviewer for his/her helpful comments and suggestions.