In the mathematical theory of probability, the entropy rate or source information rate is a function assigning an entropy to a stochastic process.
For a strongly stationary process, the conditional entropy for latest random variable eventually tend towards this rate value.
with a countable index gives rise to the sequence of its joint entropies
If the limit exists, the entropy rate is defined as Note that given any sequence
The entropy rate thus computes the mean of the first
The behaviour of joint entropies from one index to the next is also explicitly subject in some characterizations of entropy.
may be understood as a sequence of random variables, the entropy rate
represents the average entropy change per one random variable, in the long term.
It can be thought of as a general property of stochastic sources - this is the subject of the asymptotic equipartition property.
A stochastic process also gives rise to a sequence of conditional entropies, comprising more and more random variables.
For strongly stationary stochastic processes, the entropy rate equals the limit of that sequence The quantity given by the limit on the right is also denoted
, which is motivated to the extent that here this is then again a rate associated with the process, in the above sense.
Since a stochastic process defined by a Markov chain that is irreducible, aperiodic and positive recurrent has a stationary distribution, the entropy rate is independent of the initial distribution.
For example, consider a Markov chain defined on a countable number of states.
Given its right stochastic transition matrix
and an entropy associated with each state, one finds where
The entropy rate of hidden Markov models (HMM) has no known closed-form solution.
However, it has known upper and lower bounds.
Let the underlying Markov chain
, both sides converge to the middle.
[1] The entropy rate may be used to estimate the complexity of stochastic processes.
It is used in diverse applications ranging from characterizing the complexity of languages, blind source separation, through to optimizing quantizers and data compression algorithms.
For example, a maximum entropy rate criterion may be used for feature selection in machine learning.