Evidence lower bound

In statistics, the evidence lower bound (ELBO, also variational lower bound or negative variational free energy) is the quantity optimized in Variational Bayesian methods. These methods handle cases where a distribution $Q$ over unobserved variables $\mathbf {Z}$ is optimized as an approximation to the true posterior $P(\mathbf {Z} |\mathbf {X} )$, given observed data $\mathbf {X}$. Then the evidence lower bound is defined as:

$L(X)=H(Q;P(X,Z))-H(Q)=\sum _{\mathbf {Z} }Q(\mathbf {Z} )\log P(\mathbf {Z} ,\mathbf {X} )-\sum _{\mathbf {Z} }Q(\mathbf {Z} )\log Q(\mathbf {Z} )$

where $H(Q;P(X,Z))$ is cross-entropy. Maximizing the evidence lower bound minimizes $D_{\mathrm {KL} }(Q\parallel P)$, the Kullback–Leibler divergence, a measure of dissimilarity of $Q$ from the true posterior. The primary reason why this quantity is preferred for optimization is that it can be computed without access to the posterior, given a good choice of $Q$.

For other measures of dissimilarity to be optimized to fit $Q$ see Divergence (statistics).

Justification as a lower bound on the evidence

The name evidence lower bound is justified by analyzing a decomposition of the KL-divergence between the true posterior and $Q$:

${\begin{aligned}D_{\mathrm {KL} }(Q\parallel P(Z|X))&=\sum _{\mathbf {Z} }Q(\mathbf {Z} )\left[\log {\frac {Q(\mathbf {Z} )P(\mathbf {X} )}{P(\mathbf {Z} ,\mathbf {X} )}}\right]\\D_{\mathrm {KL} }(Q\parallel P)&=\sum _{\mathbf {Z} }Q(\mathbf {Z} )\left[\log {\frac {Q(\mathbf {Z} )}{P(\mathbf {Z} ,\mathbf {X} )}}+\log P(\mathbf {X} )\right]\\D_{\mathrm {KL} }(Q\parallel P)&=\sum _{\mathbf {Z} }Q(\mathbf {Z} )\log Q(\mathbf {Z} )-\sum _{\mathbf {Z} }Q(\mathbf {Z} )\log P(\mathbf {Z} ,\mathbf {X} )+\log P(\mathbf {X} )\\\log P(\mathbf {X} )-D_{\mathrm {KL} }(Q\parallel P)&=\sum _{\mathbf {Z} }Q(\mathbf {Z} )\log P(\mathbf {Z} ,\mathbf {X} )-\sum _{\mathbf {Z} }Q(\mathbf {Z} )\log Q(\mathbf {Z} )=L(X)\end{aligned}}$

As $D_{\mathrm {KL} }(Q\parallel P)\geq 0$ this equation shows that the evidence lower bound is indeed a lower bound on the log-evidence $\log P(\mathbf {X} )$ for the model considered. As $\log P(\mathbf {X} )$ does not depend on $Q$ this equation additionally shows that maximizing the evidence lower bound on the right minimizes $D_{\mathrm {KL} }(Q\parallel P)$, as claimed above.