Rényi entropy

The Rényi entropy is named after Alfréd Rényi, who looked for the most general way to quantify information while preserving additivity for independent events.

[1][2] In the context of fractal dimension estimation, the Rényi entropy forms the basis of the concept of generalized dimensions.

[3] The Rényi entropy is important in ecology and statistics as index of diversity.

The Rényi entropy is also important in quantum information, where it can be used as a measure of entanglement.

In the Heisenberg XY spin chain model, the Rényi entropy as a function of α can be calculated explicitly because it is an automorphic function with respect to a particular subgroup of the modular group.

[4][5] In theoretical computer science, the min-entropy is used in the context of randomness extractors.

is a discrete random variable with possible outcomes in the set

Applications often exploit the following relation between the Rényi entropy and the α-norm of the vector of probabilities: Here, the discrete probability distribution

⁠, the Rényi entropy is just the logarithm of the size of the support of X.

approaches infinity, the Rényi entropy is increasingly determined by the events of highest probability.

[6] If the probabilities are all nonzero, it is simply the logarithm of the cardinality of the alphabet (⁠

The collision entropy is related to the index of coincidence.

It is the negative logarithm of the Simpson diversity index.

is the largest real number b such that all events occur with probability at most ⁠

In this sense, it is the strongest way to measure the information content of a discrete random variable.

The min-entropy has important applications for randomness extractors in theoretical computer science: Extractors are able to extract randomness from random sources that have a large min-entropy; merely having a large Shannon entropy does not suffice for this task.

⁠, which can be proven by differentiation,[8] as which is proportional to Kullback–Leibler divergence (which is always non-negative), where ⁠

We can define the Rényi divergence for the special values α = 0, 1, ∞ by taking a limit, and in particular the limit α → 1 gives the Kullback–Leibler divergence.

For any fixed distributions P and Q, the Rényi divergence is nondecreasing as a function of its order α, and it is continuous on the set of α for which it is finite,[13] or for the sake of brevity, the information of order α obtained if the distribution P is replaced by the distribution Q.

[1] A pair of probability distributions can be viewed as a game of chance in which one of the distributions defines official odds and the other contains the actual probabilities.

Knowledge of the actual probabilities allows a player to profit from the game.

The expected profit rate is connected to the Rényi divergence as follows[14] where

is the distribution defining the official odds (i.e. the "market") for the game,

⁠), the long-term realized rate converges to the true expectation which has a similar mathematical structure[14] The value ⁠

The other Rényi divergences satisfy the criteria of being positive and continuous, being invariant under 1-to-1 co-ordinate transformations, and of combining additively when A and X are independent, so that if p(A, X) = p(A)p(X), then and The stronger properties of the

The Rényi entropies and divergences for an exponential family admit simple expressions[15] and where is a Jensen difference divergence.

The Rényi entropy in quantum physics is not considered to be an observable, due to its nonlinear dependence on the density matrix.

(This nonlinear dependence applies even in the special case of the Shannon entropy.)

It can, however, be given an operational meaning through the two-time measurements (also known as full counting statistics) of energy transfers[citation needed].

The limit of the quantum mechanical Rényi entropy as

Rényi entropy of a random variable with two possible outcomes against p 1 , where P = ( p 1 , 1 − p 1 ) . Shown are Η 0 , Η 1 , Η 2 and Η , with the unit on the vertical axis being the shannon .