Generalized linear models (GLMs) are widely used for data analysis in econometrics and statistics [2]-[6]. In a GLM, a location parameter (usually the expected value or mean) of a dependent variable is linked to a linear combination of independent variables through a link function. The choice of link function is determined by either the underlying error distribution (such as logistic, Poisson, gamma, or inverse-gamma) or the observational structure (such as censoring or truncation).

In contrast, activation functions are used in deep learning (DL) models [7][8]. Certain DL architectures typically consist of two types of layers: convolutional and pooling. Nawata [9][10] proposed the grouping-adjusted median estimator (GAME). However, GAME focuses mainly on the pooling stage, and activation functions in the convolutional stage have not been fully examined. Common activation functions include the step (or indicator) function used in threshold models [11]-[14], the sigmoid function [15]-[18], the rectified linear unit (ReLU) [19]-[24], and the leaky ReLU [25]-[29]. Although the universal approximation theorem [30]-[33] shows that most functions can be approximated using these activation functions in DL models, several major challenges remain, including overfitting, multiple local minima, and substantial computational requirements.

These issues become particularly serious in large language models (LLMs), which may contain trillions of unknown parameters due to their numerous layers and nodes (neurons). Selecting an appropriate activation function is therefore critical. Link functions and activation functions are closely related and often coincide. In this note, we examine the similarities between these functions and propose an alternative activation function derived from the conditional expectation (truncated mean) of a truncated model. Our focus is on the theoretical and methodological aspects rather than practical applications.

In this note, we consider the expected value (hereafter, the mean) and other location parameters, such as the median. Following Nawata [34], the GLM is specified as

y i * is an unobserved latent variable, and y i is an observed dependent variable. x i and β are vectors of independent variables (including a constant term) and unknown parameters, respectively, and u i is an error term. g ( z ) is a monotonically increasing function such that g ( z 1 ) ≥ g ( z 2 ) if z 1 ≥ z 2 . If it is not uniformly monotonically increasing, we consider a local interval in which g ( z ) is monotonic. If it is monotonically decreasing, we use − g ( z ) . Let ℒ denote a function that gives a suitable location parameter of y i .

Usually, we consider the mean,

where E u denotes the expected value with respect to u i , and ℒ depends on the distribution of the error term. However, when the median is used as the location parameter, we obtain

M e d denotes the median with respect to the error term. The result does not depend on the distribution of the error term, provided that its median is zero. In this sense, the estimator is robust.

In the binary response model, y i takes the value 0 or 1 and is given by y i = g ( y i * ) = 1 ( x ′ i β + u i > 0 ) . We obtain

Here, 1 ( A ) is the indicator function, which takes the value 1 if A is true and 0 otherwise, and F is the distribution function of − u i . Assuming a distribution symmetric about zero, we treat F as the distribution function of u i itself. Under the standard logistic distribution,

we obtain the logit model.

When the median is used, the model becomes semiparametric. No specific distribution function is required, and the resulting estimator is robust. However, Nawata [35] showed that such robust estimators cannot be n -consistent and are less efficient than the maximum likelihood estimator (MLE) in the binary response model.

These link functions are similar to the activation functions h ( z ) used in DL. The step function in threshold models is equivalent to the indicator function in the median case. However, threshold models and their corresponding activation function have a major drawback: the derivative is zero except at the jump discontinuity, causing backpropagation to fail and making optimization difficult. This is consistent with the inefficiency of the median estimator in the binary response model. The sigmoid function corresponds to the logistic distribution function. Even when the sigmoid function is used as an activation function, the vanishing gradient problem—in which the derivative approaches zero—makes learning extremely slow and computationally expensive.

For censored regression or Tobit models, g ( z ) = max { 0 , z } and

The MLE, which assumes a specific distribution for the error term (usually normal or logistic), is widely used in these models. However, it is inconsistent when the distribution is misspecified. Powell [36] considers the least absolute deviations estimator (PLAD), which minimizes

PLAD is the GLM estimator that uses the median as the location parameter and is consistent when the median of the error term is zero. Its drawbacks are that (i) multiple local minima make the estimator difficult to compute (Nawata [37]) and (ii) its asymptotic variance is much larger than that of the MLE, making it inefficient.

In DL, the ReLU function,

is widely used as an activation function, corresponding to the GLM link function when the error term has a median of zero. Its derivative satisfies ∂ h ( z ) / ∂ z = 1 if z > 0 ∂ h ( z ) / ∂ z = 0 if z < 0 ; the function is not differentiable at z = 0 . This means that (i) information from observations with x ′ i β < 0 is used only through their sign, (ii) the first derivative is discontinuous and jumps at zero, and (iii) the second derivative is zero at any differentiable point. Lin et al. ([38], p. 2) state “ReLU-based functions that eliminate negative values appear to constrain representational capacity in ANNs”. By analogy with PLAD, problems of multiple local minima are expected. These issues make computation difficult, especially during backpropagation. Therefore, alternatives such as sigmoid linear unit (SiLU or Swish) [39], leaky ReLU, softplus [40], and Gaussian error linear units (GELUs) [41] functions are also used, as they have been reported to improve performance [39]. The leaky ReLU function is,

where a is a small positive constant. The derivative is 1 for z > 0 and a for z < 0 . However, the modifications (including other functions previously described) are ad hoc and lack theoretical justification. Moreover, the derivative remains discontinuous at zero, and the fundamental problems of the ReLU function persist. As in the binary case, it is natural to consider the mean.

Link functions in GLMs and activation functions in the convolutional layers of DL share important similarities. In this note, we demonstrated that GLM link functions correspond to the activation functions used in binary models. The step function in threshold models is equivalent to the indicator function in the median case. The sigmoid function represents the mean of the indicator function under the assumption that the error term in the GLM follows a logistic distribution.

When the ReLU function is used, the model corresponds to the median regression model in GLMs. Its disadvantages include the fact that observations in the negative region have no effect and its first derivative takes only the values zero or one, complicating optimization. Powell [36] proposed the least absolute deviations estimator, but estimation remains difficult because of differentiability issues and multiple local minima. Although the leaky ReLU function is also used, it lacks theoretical justification, and computational challenges persist.

To address these issues, we propose using the TMF as an alternative activation function, derived by applying a truncated normal model. The TMF is widely used in GLMs, such as in Heckman’s two-step estimator [43][44]. It is infinitely differentiable, and its second derivative is non-zero at any point, which may facilitate optimization. It is also well approximated by the leaky ReLU function.

The relationship between link functions and activation functions remains an open question. Evaluating the performance of the TMF in DL models is both necessary and compelling, representing an important direction for future research that will be highly valuable for advancing DL methodology. In this note, we focus on the theoretical and methodological aspects, leaving practical applications for future work.

The author would like to thank an anonymous referee for his/her helpful comments and suggestions.