Consider a random experiment with n different outcomes, with the corresponding probabilities { p i } i = 1 n with ∑ i = 1 n p i = 1 . For such an experiment, the discrete Shannon entropy, also known as the information theoretic entropy S is defined by [1]-[3]

Here we restrict ourselves to the natural logarithm, but in general, Shannon entropy can have the logarithm of any base. If all possible outcomes are equally likely, Shannon entropy is also related to thermodynamic configurational entropy, or Boltzmann entropy, by [4][5]

where k B is the Boltzmann constant and Ω is the number of possible outcomes or microstates.

For a continuous random variable x , the common practice is to replace the discrete probabilities p i in Equation (1) by the probability density function of the variable f ( x ) , and change the summation to integral. Therefore, the continuous Shannon entropy, also known as differential entropy, for the continuous random variable, is written as [6]-[8]

where the limits a and b define the support set or the interval of the random variable x . This extension of Shannon entropy, however, suffers from certain shortcomings. For example, it is dimensionally inconsistent. Thus, if the random variable has a physical dimension such as length L, then the dimension of f ( x ) will be L⁻¹. However, the argument of a logarithmic function must be dimensionless. More specifically, if the variable x is length, depending on whether we use centimeters or meters for x , the value of the entropy changes. Another shortcoming is that for certain functions f ( x ) , the entropy becomes negative whereas the discrete Shannon entropy is always non-negative. For example, for the probability density function

the continuous Shannon entropy of Equation (3) becomes negative. Nevertheless, continuous Shannon entropy has been applied in many areas, including statistical mechanics, thermodynamics, as well as signal processing and information theory.

Some of the inconsistencies described above have been pointed out in the literature [9]. In this article, we re-examine the transition from discrete to continuous Shannon entropy in a rigorous way. We clearly show that the continuous Shannon entropy described by Equation (3) is not the direct limit of the discrete entropy.

Let f ( x ) be the probability density function of a random variable x defined on the interval [ a , b ] . Then the probability of the outcome being between x and x + Δ x is approximately f ( x ) Δ x , which becomes exact when Δ x → 0 .

Let us divide the interval [ a , b ] into n equal slices, each of width

Then the discrete Shannon entropy of the outcomes would be

which can be written as

where S n indicates approximation of S when f ( x ) is discretized by n terms. Now, if we let n → ∞ such that Δ x i → 0 , then each term in the second sum vanishes since lim u → 0 u ln u = 0 . The first sum, on the other hand, by definition becomes a Riemann integral,

Therefore, it seems plausible that the discrete Shannon entropy reduces to the continuous entropy given by Equation (3).

The problem, however, is that if we compare the corresponding terms in the two sums of Equation (7), we see that as Δ x i → 0 , the terms of the second sum are much larger than those in the first sum. This is because the ratio of a term in the second sum to the corresponding term in the first sum is

Then as Δ x i → 0 , the denominator of this fraction remains finite whereas the numerator approaches − ∞ . Consequently, in the above transition from discrete to continuous Shannon entropy, the main term is discarded. This accounts for the shortcomings of the continuous Shannoon entropy given by Equation (3).

We now take a closer look at a more fundamental question: Can the definition of Shannon entropy be extended to continuous random variables as described above? To do so, we start with Equation (6), which is the correct Shannon entropy for a discretized continuous probability distribution. Using Equation (5), this equation becomes

which reduces to

or

where the over-bar represents average value. Now, as n → ∞ , every term on the

right hand side of this expression remains finite except ln ( b − a n ) which becomes − ∞ . Therefore,

Consequently, as stated earlier, differential entropy is not the direct limit of discrete entropy [9]. We also verified this result computationally.

Although continuous Shannon entropy is applied in many areas, it lacks some of the properties of entropy. Extending the expression of discrete Shannon entropy to a continuous random variable results in an infinite entropy; therefore, such a simple extension is not possible. Furthermore, Equation (3), which is commonly used to express differential or continuous entropy is dimensionally inconsistent, scale dependent, and for certain probability density functions, becomes negative. Despite the fact that its shortcomings have been pointed out in some references, the expression continues to be applied in many fields.

It is also worth pointing out that some authors have suggested an alternative definition for continuous entropy by introducing an invariant factor m ( x ) into Equation (3) [7],

However, because of this added function, this equation should not be called continuous “Shannon” entropy.