Log sum inequality

From Wikipedia, the free encyclopedia
Jump to: navigation, search


In mathematics, the log sum inequality is an inequality which is useful for proving several theorems in information theory.

Statement[edit]

Let and be nonnegative numbers. Denote the sum of all s by and the sum of all s by . The log sum inequality states that

with equality if and only if are equal for all .

Proof[edit]

Notice that after setting we have

where the inequality follows from Jensen's inequality since , , and is convex.

Applications[edit]

The log sum inequality can be used to prove several inequalities in information theory such as Gibbs' inequality or the convexity of Kullback-Leibler divergence.

For example, to prove Gibbs' inequality it is enough to substitute s for s, and s for s to get

Generalizations[edit]

The inequality remains valid for provided that and . The proof above holds for any function such that is convex, such as all continuous non-decreasing functions. Generalizations to convex functions other than the logarithm is given in Csiszár, 2004.

References[edit]