Log sum inequality

From Wikipedia, the free encyclopedia
Jump to: navigation, search

In mathematics, the log sum inequality is an inequality which is useful for proving several theorems in information theory.


Let and be nonnegative numbers. Denote the sum of all s by and the sum of all s by . The log sum inequality states that

with equality if and only if are equal for all .


Notice that after setting we have

where the inequality follows from Jensen's inequality since , , and is convex.


The log sum inequality can be used to prove several inequalities in information theory such as Gibbs' inequality or the convexity of Kullback-Leibler divergence.

For example, to prove Gibbs' inequality it is enough to substitute s for s, and s for s to get


The inequality remains valid for provided that and . The proof above holds for any function such that is convex, such as all continuous non-decreasing functions. Generalizations to convex functions other than the logarithm is given in Csiszár, 2004.