In 2010, Wu and Verdú gave an operational characterization of Rényi information dimension as the fundamental limit of almost lossless data compression for analog sources under various regularity constraints of the encoder/decoder.
If the information dimension exists, one can define the -dimensional entropy of this distribution by
provided the limit exists. If , the zero-dimensional entropy equals the standard Shannon entropy. For integer dimension , the -dimensional entropy is the -fold integral defining the respective differential entropy.
where and ; is a purely atomic probability measure (discrete part), is the absolutely continuous probability measure, and is a probability measure singular with respect to Lebesgue measure but with no atoms (singular part). Let be a random variable such that . Assume the distribution of can be represented as
where is a discrete measure and is the absolutely continuous probability measure with . Then
Moreover, given and differential entropy , the -Dimensional Entropy is simply given by
where is the Shannon entropy of a discrete random variable with and and given by
It is shown  that information dimension and differential entropy are tightly connected.
Let be a positive random variable with density .
Suppose we divide the range of into bins of length . By the mean value theorem, there exists a value within each bin such that
Consider the discretized random variable if .
The probability of each support point is
The entropy of this variable is
If we set and then we are doing exactly the same quantization as the definition of information dimension. Since relabeling the events of a discrete random variable does not change its entropy, we have
and when is sufficient large,
which is the differential entropy of the continuous random variable. In particular, if is Riemann integrable, then
Comparing this with the -dimensional entropy shows that the differential entropy is exactly the one-dimensional entropy
In fact, this can be generalized to higher dimensions. Rényi shows that, if is a random vector in a -dimensional Euclidean space with an absolutely continuous distribution with a probability density function and finite entropy of the integer part (), we have
The information dimension of a distribution gives a theoretical upper bound on the compression rate, if we want to compress a variable coming from this distribution. In the context of lossless data compression, we try to compress real number with less real number which both have infinite precision.
The main objective of the lossless data compression is to find efficient representations for source realizations by . A code for is a pair of mappings:
encoder: which converts information from a source into symbols for communication or storage;
decoder: is the reverse process, converting code symbols back into a form that the recipient understands.
The block error probability is .
Define to be the infimum of such that there exists a sequence of codes such that for all sufficiently large .
So basically gives the ratio between the code length and the source length, it shows how good a specific encoder decoder pair is. The fundamental limits in lossless source coding are as follows.
Consider a continuous encoder function with its continuous decoder function . If we impose no regularity on and , due to the rich structure of , we have the minimum -achievable rate for all . It means that we can built an encoder/decoder pair with infinity compression rate.
In order to get some nontrivial and meaningful conclusions, let the minimum achievable rate for linear encoder and Borel decoder. If random variable has a distribution which is a mixture of discrete and continuous part. Then for all Suppose we restrict the decoder to be a Lipschitz continuous function and holds, then the minimum achievable rate for all .