User:K687 ypo/sandbox

Basic examples

</ref>

Kullback gives the following example (Table 2.1, Example 2.1). Let P and Q be the distributions shown in the table and figure. P is the distribution on the left side of the figure, a binomial distribution with N = 2 and p = 0.4. Q is the distribution on the right side of the figure, a discrete uniform distribution with the three possible outcomes x = 0, 1, or 2, each with probability p = 1/3.

	0	1	2
Distribution P	0.36	0.48	0.16
Distribution Q	0.333	0.333	0.333

The KL divergence is calculated as follows. This example uses the natural log with base e, designated ln().

$D_{\mathrm {KL} }(Q\|P)=\sum _{i}Q(i)\ln {\frac {Q(i)}{P(i)}}$

$=0.333*\ln {\frac {0.333}{0.36}}+0.333*\ln {\frac {0.333}{0.48}}+0.333*\ln {\frac {0.333}{0.16}}$

$=-0.02565+(-0.12155)+0.24466$

$=0.09746$