Rényi entropy

From Wikipedia, the free encyclopedia
  (Redirected from Renyi divergence)
Jump to: navigation, search

In information theory, the Rényi entropy generalizes the Hartley entropy, the Shannon entropy, the collision entropy and the min-entropy. Entropies quantify the diversity, uncertainty, or randomness of a system. The Rényi entropy is named after Alfréd Rényi.[1]

The Rényi entropy is important in ecology and statistics as indices of diversity. The Rényi entropy is also important in quantum information, where it can be used as a measure of entanglement. In the Heisenberg XY spin chain model, the Rényi entropy as a function of α can be calculated explicitly by virtue of the fact that it is an automorphic function with respect to a particular subgroup of the modular group.[2][3] In theoretical computer science, the min-entropy is used in the context of randomness extractors.


The Rényi entropy of order \alpha, where \alpha \geq 0 and \alpha \neq 1 , is defined as

H_\alpha(X) = \frac{1}{1-\alpha}\log\Bigg(\sum_{i=1}^n p_i^\alpha\Bigg) .[1]

Here, X is a discrete random variable with possible outcomes 1,2,...,n and corresponding probabilities p_i \doteq \Pr(X=i) for i=1,\dots,n, and the logarithm is base 2. If the probabilities are p_i=1/n for all i=1,\dots,n, then all the Rényi entropies of the distribution are equal: H_\alpha(X)=\log n. In general, for all discrete random variables X, H_\alpha(X) is a non-increasing function in \alpha.

Applications often exploit the following relation between the Rényi entropy and the p-norm of the vector of probabilities:

H_\alpha(X)=\frac{\alpha}{1-\alpha} \log \left(\|P\|_\alpha\right) .

Here, the discrete probability distribution P=(p_1,\dots,p_n) is interpreted as a vector in \R^n with p_i\geq 0 and \sum_{i=1}^{n} p_i =1.

The Rényi entropy for any \alpha \geq 0 is Schur concave.

Special cases of the Rényi entropy[edit]

Rényi entropy of a random variable with two possible outcomes against p1, where P = (p1, 1 − p1). Shown are H0, H1, H2 and H, in units of shannons.

As \alpha approaches zero, the Rényi entropy increasingly weighs all possible events more equally, regardless of their probabilities. In the limit for \alpha\to 0, the Rényi entropy is just the logarithm of the size of the support of X. The limit for \alpha\to 1 is the Shannon entropy. As \alpha approaches infinity, the Rényi entropy is increasingly determined by the events of highest probability.

Hartley or max-entropy[edit]

Provided the probabilities are nonzero,[4] H_0 is the logarithm of the cardinality of X, sometimes called the Hartley entropy of X:

H_0 (X) = \log n = \log |X|.\,

Shannon entropy[edit]

The limiting value of H_\alpha as \alpha \rightarrow 1 is the Shannon entropy:[5]

H_1 (X) = - \sum_{i=1}^n p_i \log p_i.

Collision entropy[edit]

Collision entropy, sometimes just called "Rényi entropy," refers to the case \alpha = 2,

H_2 (X) = - \log \sum_{i=1}^n p_i^2 = - \log P(X = Y)

where X and Y are independent and identically distributed.


Main article: Min entropy

In the limit as \alpha \rightarrow \infty , the Rényi entropy H_\alpha converges to the min-entropy H_\infty:

H_\infty(X) \doteq \min_i (-\log p_i) = -(\max_i \log p_i) = -\log \max_i p_i\,.

Equivalently, the min-entropy H_\infty(X) is the largest real number b such that all events occur with probability at most 2^{-b}.

The name min-entropy stems from the fact that it is the smallest entropy measure in the family of Rényi entropies. In this sense, it is the strongest way to measure the information content of a discrete random variable. In particular, the min-entropy is never larger than the Shannon entropy.

The min-entropy has important applications for randomness extractors in theoretical computer science: Extractors are able to extract randomness from random sources that have a large min-entropy; merely having a large Shannon entropy does not suffice for this task.

Inequalities between different values of α[edit]

That H_\alpha is non-increasing in \alpha, which can be proven by differentiation,[6] as

-\frac{d H_\alpha}{d\alpha}
= \frac{1}{(1-\alpha)^2} \sum_{i=1}^n z_i \log(z_i / p_i),

which is proportional to Kullback–Leibler divergence (which is always non-negative), where z_i = p_i^\alpha / \sum_{j=1}^n p_j^\alpha.

In particular cases inequalities can be proven also by Jensen's inequality:[7][8]

\log n=H_0\geq H_1 \geq H_2 \geq H_\infty .

For values of \alpha>1, inequalities in the other direction also hold. In particular, we have[9][citation needed]

 H_2 \le 2H_\infty .

On the other hand, the Shannon entropy H_1 can be arbitrarily high for a random variable X that has a given min-entropy.[citation needed]

Rényi divergence[edit]

As well as the absolute Rényi entropies, Rényi also defined a spectrum of divergence measures generalising the Kullback–Leibler divergence.[10]

The Rényi divergence of order α or alpha-divergence of a distribution P from a distribution Q is defined to be:

D_\alpha (P \| Q) = \frac{1}{\alpha-1}\log\Bigg(\sum_{i=1}^n \frac{p_i^\alpha}{q_i^{\alpha-1}}\Bigg) \,

when 0 < \alpha < \infty and \alpha \ne 1. We can define the Rényi divergence for the special values \alpha = 0, 1, \infty by taking a limit, and in particular the limit \alpha \to 1 gives the Kullback-Leibler divergence.

Some special cases:

D_0(P \| Q) = - \log Q(\{i : p_i > 0\}) : minus the log probability under Q that pi > 0;
D_{1/2}(P \| Q) = -2 \log \sum_{i=1}^n \sqrt{p_i q_i}  : minus twice the logarithm of the Bhattacharyya coefficient; (Nielsen & Boltz (2009))
D_1(P \| Q) = \sum_{i=1}^n p_i \log \frac{p_i}{q_i}  : the Kullback-Leibler divergence;
D_2(P \| Q) = \log \Big\langle \frac{p_i}{q_i} \Big\rangle  : the log of the expected ratio of the probabilities;
D_\infty(P \| Q) = \log \sup_i \frac{p_i}{q_i}  : the log of the maximum ratio of the probabilities.

The Rényi divergence is indeed a divergence, meaning simply that D_\alpha (P \| Q) is greater than or equal to zero, and zero only when P = Q. For any fixed distributions P and Q, the Rényi divergence is nondecreasing as a function of its order α, and it is continuous on the set of α for which it is finite.[10]

Why α=1 is special[edit]

The value α = 1, which gives the Shannon entropy and the Kullback–Leibler divergence, is special because it is only at α=1 that the chain rule of conditional probability holds exactly:

H(A,X) =  H(A) + \mathbb{E}_{a \sim A} \big[ H(X| A=a) \big]

for the absolute entropies, and

D_\mathrm{KL}(p(x|a)p(a)||m(x,a)) =  D_\mathrm{KL}(p(a)||m(a)) + \mathbb{E}_{p(a)}\{D_\mathrm{KL}(p(x|a)||m(x|a))\},

for the relative entropies.

The latter in particular means that if we seek a distribution p(x,a) which minimizes the divergence from some underlying prior measure m(x,a), and we acquire new information which only affects the distribution of a, then the distribution of p(x|a) remains m(x|a), unchanged.

The other Rényi divergences satisfy the criteria of being positive and continuous; being invariant under 1-to-1 co-ordinate transformations; and of combining additively when A and X are independent, so that if p(A,X) = p(A)p(X), then

H_\alpha(A,X) = H_\alpha(A) + H_\alpha(X)\;


D_\alpha(P(A)P(X)\|Q(A)Q(X)) = D_\alpha(P(A)\|Q(A)) + D_\alpha(P(X)\|Q(X)).

The stronger properties of the α = 1 quantities, which allow the definition of conditional information and mutual information from communication theory, may be very important in other applications, or entirely unimportant, depending on those applications' requirements.

Exponential families[edit]

The Rényi entropies and divergences for an exponential family admit simple expressions [11]

H_\alpha(p_F(x;\theta)) =  \frac{1}{1-\alpha} \left(F(\alpha\theta)-\alpha F(\theta)+\log E_p[e^{(\alpha-1)k(x)}]\right)


D_\alpha(p:q) = \frac{J_{F,\alpha}(\theta:\theta')}{1-\alpha}


J_{F,\alpha}(\theta:\theta')= \alpha F(\theta)+(1-\alpha) F(\theta')- F(\alpha\theta+(1-\alpha)\theta')

is a Jensen difference divergence.

Physical meaning[edit]

Renyi entropy in quantum physics is not considered an observable, due to its nonlinear dependence on density matrix. The Shannon entropy shares this nonlinear dependence. Recently, Nazarov showed a correspondence that reveals the physical meaning of the Renyi entropy flow in time. His proposal is similar to the fluctuation-dissipation theorem in spirit and allows to measure quantum entropy using full counting statistics (FCS) of energy transfers. [12]

See also[edit]


  1. ^ a b Rényi (1961)
  2. ^ Franchini (2008)
  3. ^ Its (2010)
  4. ^ RFC 4086, page 6
  5. ^ Bromiley, Thacker & Bouhova-Thacker (2004)
  6. ^ Beck (1993)
  7. ^  H_1 \ge H_2 holds because  \sum\limits_{i = 1}^M {p_i \log p_i }  \le \log \sum\limits_{i = 1}^M {p_i^2 } .
  8. ^  H_\infty \le H_2 holds because 
\log \sum\limits_{i = 1}^n {p_i^2 }  \le \log \sup _i p_i \left( {\sum\limits_{i = 1}^n {p_i } } \right) = \log \sup p_i 
  9. ^  H_2 \le 2H_\infty holds because  \log \sum\limits_{i = 1}^n {p_i^2 }  \ge \log \sup _i p_i^2  = 2\log \sup_i p_i
  10. ^ a b Van Erven, Tim; Harremoës, Peter (2014). "Rényi Divergence and Kullback-Leibler Divergence". IEEE Transactions on Information Theory 60 (7): 3797–3820. doi:10.1109/TIT.2014.2320500. 
  11. ^ Nielsen & Nock (2011)
  12. ^ Nazarov, Y. (2011). "Flows of Renyi entropies,". Physical Review B 84: 205437. arXiv:1408.3910. Bibcode:2015PhRvB..91j4303A. doi:10.1103/PhysRevB.91.104303. , Nazarov, Y. (2015). "Renyi entropy flows from quantum heat engines,". Physical Review B 91: 104303. arXiv:1408.3910. Bibcode:2015PhRvB..91j4303A. doi:10.1103/PhysRevB.91.104303. , Nazarov, Y. (2015). "Exact correspondence between Renyi entropy flows and physical flows,". Physical Review B 91: 174307. doi:10.1103/PhysRevB.91.174307. 


  • Beck, Christian; Schlögl, Friedrich (1993). Thermodynamics of chaotic systems: an introduction. Cambridge University Press. ISBN 0521433673. 
  • Nielsen, F.; Boltz, S. (2010). "The Burbea-Rao and Bhattacharyya centroids". arXiv:1004.5049.