Pinsker's inequality

From Wikipedia, the free encyclopedia
Jump to: navigation, search

In information theory, Pinsker's inequality, named after its inventor Mark Semenovich Pinsker, is an inequality that bounds the total variation distance (or statistical distance) in terms of the Kullback–Leibler divergence. The inequality is tight up to constant factors.[1]

Formal statement[edit]

Pinsker's inequality states that, if and are two probability distributions on a measurable space , then


is the total variation distance (or statistical distance) between and and

is the Kullback–Leibler divergence in nats. When the sample space is a finite set, the Kullback–Leibler divergence is given by

Note that in terms of the total variation norm of the signed measure , Pinsker's inequality differs from the one given above by a factor of two:

The proof of Pinsker's inequality uses the partition inequality for f-divergences.


Pinsker first proved the inequality with a worse constant. The inequality in the above form was proved independently by Kullback, Csiszár, and Kemperman.[2]

Inverse problem[edit]

A precise inverse of the inequality cannot hold: for every , there are distributions with but .[3]


  1. ^ Csiszár, Imre; Körner, János (2011). Information Theory: Coding Theorems for Discrete Memoryless Systems. Cambridge University Press. p. 44. ISBN 9781139499989. 
  2. ^ Tsybakov, Alexandre (2009). Introduction to Nonparametric Estimation. Springer. p. 132. ISBN 9780387790527. 
  3. ^ The divergence becomes infinite whenever one of the two distributions assigns probability zero to an event while the other assigns it a nonzero probability (no matter how small); see e.g. Basu, Mitra; Ho, Tin Kam (2006). Data Complexity in Pattern Recognition. Springer. p. 161. ISBN 9781846281723. .

Additional reading[edit]

  • Thomas M. Cover and Joy A. Thomas: Elements of Information Theory, 2nd edition, Willey-Interscience, 2006
  • Nicolo Cesa-Bianchi and Gábor Lugosi: Prediction, Learning, and Games, Cambridge University Press, 2006