Entropy power inequality
In information theory, the entropy power inequality is a result that relates to so-called "entropy power" of random variables. It shows that the entropy power of suitably well-behaved random variables is a superadditive function. The entropy power inequality was proved in 1948 by Claude Shannon in his seminal paper "A Mathematical Theory of Communication". Shannon also provided a sufficient condition for equality to hold; Stam (1959) showed that the condition is in fact necessary.
Statement of the inequality
and the entropy power of X, denoted N(X), is defined to be
In particular, N(X) = |K| 1/n when X is normal distributed with covariance matrix K.
- Information entropy
- Information theory
- Limiting density of discrete points
- Kullback–Leibler divergence
- Entropy estimation
- Dembo, Amir; Cover, Thomas M.; Thomas, Joy A. (1991). "Information-theoretic inequalities". IEEE Trans. Inform. Theory. 37 (6): 1501–1518. doi:10.1109/18.104312. MR 1134291.
- Gardner, Richard J. (2002). "The Brunn–Minkowski inequality". Bull. Amer. Math. Soc. (N.S.). 39 (3): 355–405 (electronic). doi:10.1090/S0273-0979-02-00941-2.
- Shannon, Claude E. (1948). "A mathematical theory of communication". Bell System Tech. J. 27: 379–423, 623–656.
- Stam, A. J. (1959). "Some inequalities satisfied by the quantities of information of Fisher and Shannon". Information and Control. 2 (2): 101–112. doi:10.1016/S0019-9958(59)90348-1.