Differential entropy (also referred to as continuous entropy) is a concept in information theory that began as an attempt by Claude Shannon to extend the idea of (Shannon) entropy (a measure of average surprisal) of a random variable, to continuous probability distributions.
Unfortunately, Shannon did not derive this formula, and rather just assumed it was the correct continuous analogue of discrete entropy, but it is not.
Differential entropy (described here) is commonly encountered in the literature, but it is a limiting case of the LDDP, and one that loses its fundamental association with discrete entropy.
be a random variable with a probability density function
For probability distributions which do not have an explicit density function expression, but have an explicit quantile function expression,
as[3]: 54–59 As with its discrete analog, the units of differential entropy depend on the base of the logarithm, which is usually 2 (i.e., the units are bits).
Related concepts such as joint, conditional differential entropy, and relative entropy are defined in a similar fashion.
Unlike the discrete analog, the differential entropy has an offset that depends on the units used to measure
One must take care in trying to apply properties of discrete entropy to differential entropy, since probability density functions can be greater than 1.
has the distinction of retaining its fundamental significance as a measure of discrete information since it is actually the limit of the discrete mutual information of partitions of
Thus it is invariant under non-linear homeomorphisms (continuous and uniquely invertible maps),[5] including linear[6] transformations of
, and still represents the amount of discrete information that can be transmitted over a channel that admits a continuous space of values.
For the direct analogue of discrete entropy extended to the continuous space, see limiting density of discrete points.
However, differential entropy does not have other desirable properties: A modification of differential entropy that addresses these drawbacks is the relative information entropy, also known as the Kullback–Leibler divergence, which includes an invariant measure factor (see limiting density of discrete points).
With a normal distribution, differential entropy is maximized for a given variance.
A Gaussian random variable has the largest entropy amongst all random variables of equal variance, or, alternatively, the maximum entropy distribution under constraints of mean and variance is the Gaussian.
Since differential entropy is translation invariant we can assume that
Consider the Kullback–Leibler divergence between the two distributions Now note that because the result does not depend on
When the entropy of g(x) is at a maximum and the constraint equations, which consist of the normalization condition
, are both satisfied, then a small variation δg(x) about g(x) will produce a variation δL about L which is equal to zero: Since this must hold for any small δg(x), the term in brackets must be zero, and solving for g(x) yields: Using the constraint equations to solve for λ0 and λ yields the normal distribution: Let
be an exponentially distributed random variable with parameter
, that is, with probability density function Its differential entropy is then Here,
to make it explicit that the logarithm was taken to base e, to simplify the calculation.
The differential entropy yields a lower bound on the expected squared error of an estimator.
is the beta function, and γE is Euler's constant.
For example, the differential entropy can be negative; also it is not invariant under continuous coordinate transformations.
Edwin Thompson Jaynes showed in fact that the expression above is not the correct limit of the expression for a finite set of probabilities.
[10]: 181–218 A modification of differential entropy adds an invariant measure factor to correct this, (see limiting density of discrete points).
is further constrained to be a probability density, the resulting notion is called relative entropy in information theory: The definition of differential entropy above can be obtained by partitioning the range of
Note that this procedure suggests that the entropy in the discrete sense of a continuous random variable should be