In information theory and statistics, Kullback's inequality is a lower bound on the Kullback–Leibler divergence expressed in terms of the large deviations rate function. If P and Q are probability distributions on the real line, such that P is absolutely continuous with respect to Q, i.e. P<<Q, and whose first moments exist, then
The Cramér–Rao bound is a corollary of this result.
for every measurable set A, where is the moment-generating function of Q. (Note that Q0=Q.) Then
By Gibbs' inequality we have so that
Simplifying the right side, we have, for every real θ where
Corollary: the Cramér–Rao bound
Start with Kullback's inequality
Let Xθ be a family of probability distributions on the real line indexed by the real parameter θ, and satisfying certain regularity conditions. Then
The left side of this inequality can be simplified as follows:
- where we have expanded the logarithm in a Taylor series in ,
which is half the Fisher information of the parameter θ.
The right side of the inequality can be developed as follows:
This supremum is attained at a value of t=τ where the first derivative of the cumulant-generating function is but we have so that
Putting both sides back together
which can be rearranged as:
- Kullback–Leibler divergence
- Cramér–Rao bound
- Fisher information
- Large deviations theory
- Convex conjugate
- Rate function
- Moment-generating function
Notes and references
- Fuchs, Aimé; Letta, Giorgio (1970). L'inégalité de Kullback. Application à la théorie de l'estimation. Séminaire de probabilités 4. Strasbourg. pp. 108–131.