Log sum inequality
Let and be nonnegative numbers. Denote the sum of all s by and the sum of all s by . The log sum inequality states that
with equality if and only if are equal for all .
Notice that after setting we have
where the inequality follows from Jensen's inequality since , , and is convex.
For example to prove Gibbs' inequality it is enough to substitute s for s, and s for s to get
The inequality remains valid for provided that and . Generalizations to convex functions other than the logarithm is given in Csiszár, 2004.
- T.S. Han, K. Kobayashi, Mathematics of information and coding. American Mathematical Society, 2001. ISBN 0-8218-0534-7.
- Information Theory course materials, Utah State University . Retrieved on 2009-06-14.
- Csiszár, I.; Shields, P. (2004). "Information Theory and Statistics: A Tutorial". Foundations and Trends in Communications and Information Theory 1 (4): 417–528. doi:10.1561/0100000004. Retrieved 2009-06-14.