Information dimension

[1] Simply speaking, it is a measure of the fractal dimension of a probability distribution.

It characterizes the growth rate of the Shannon entropy given by successively finer discretizations of the space.

In 2010, Wu and Verdú gave an operational characterization of Rényi information dimension as the fundamental limit of almost lossless data compression for analog sources under various regularity constraints of the encoder/decoder.

is the floor operator which converts a real number to the greatest integer less than it.

Then and are called lower and upper information dimensions of

-fold integral defining the respective differential entropy.

In 1994, Kawabata and Dembo in Kawabata & Dembo 1994 proposed a new way of measuring information based on rate distortion value of a random variable.

is the rate-distortion function that is defined as or equivalently, minimum information that could lead to a

Formally, Using the above definition of Rényi information dimension, a similar measure to d-dimensional entropy is defined in Charusaie, Amini & Rini 2022 .

that is named dimensional-rate bias is defined in a way to capture the finite term of rate-distortion function.

Formally, The dimensional-rate bias is equal to d-dimensional rate for continuous, discrete, and discrete-continuous mixed distribution.

Furthermore, it is calculable for a set of singular random variables, while d-dimensional entropy does not necessarily exist there.

According to Lebesgue decomposition theorem,[2] a probability distribution can be uniquely represented by the mixture

is a purely atomic probability measure (discrete part),

is the Shannon entropy of a discrete random variable

It is characterized by an atomic mass of weight 0.5 and has a Gaussian PDF for all

With this mixture distribution, we apply the formula above and get the information dimension

The normalized right part of the zero-mean Gaussian distribution has entropy

It is shown [3] that information dimension and differential entropy are tightly connected.

Since relabeling the events of a discrete random variable does not change its entropy, we have This yields and when

with an absolutely continuous distribution with a probability density function

The information dimension of a distribution gives a theoretical upper bound on the compression rate, if one wants to compress a variable coming from this distribution.

The main objective of the lossless data compression is to find efficient representations for source realizations

is a pair of mappings: The block error probability is

The fundamental limits in lossless source coding are as follows.

It means that one can build an encoder-decoder pair with infinity compression rate.

achievable rate for linear encoder and Borel decoder.

Suppose we restrict the decoder to be a Lipschitz continuous function and

The fundamental role of information dimension in lossless data compression further extends beyond the i.i.d.

[5] This result allows for further compression that was not possible by considering only marginal distribution of the process.