# Jensen–Shannon divergence

(Redirected from Jensen-Shannon divergence)

In probability theory and statistics, the JensenShannon divergence is a method of measuring the similarity between two probability distributions. It is also known as information radius (IRad) or total divergence to the average. It is based on the Kullback–Leibler divergence, with some notable (and useful) differences, including that it is symmetric and it always has a finite value. The square root of the Jensen–Shannon divergence is a metric often referred to as Jensen-Shannon distance.

## Definition

Consider the set $M_{+}^{1}(A)$ of probability distributions where A is a set provided with some σ-algebra of measurable subsets. In particular we can take A to be a finite or countable set with all subsets being measurable.

The Jensen–Shannon divergence (JSD) $M_{+}^{1}(A)\times M_{+}^{1}(A)\rightarrow [0,\infty {})$ is a symmetrized and smoothed version of the Kullback–Leibler divergence $D(P\parallel Q)$ . It is defined by

${\rm {JSD}}(P\parallel Q)={\frac {1}{2}}D(P\parallel M)+{\frac {1}{2}}D(Q\parallel M)$ where $M={\frac {1}{2}}(P+Q)$ The geometric Jensen–Shannon divergence (or G-Jensen–Shannon divergence) yields a closed-form formula for divergence between two Gaussian distributions by taking the geometric mean.

A more general definition, allowing for the comparison of more than two probability distributions, is:

{\begin{aligned}{\rm {JSD}}_{\pi _{1},\ldots ,\pi _{n}}(P_{1},P_{2},\ldots ,P_{n})&=\sum _{i}\pi _{i}D(P_{i}\parallel M)\\&=H\left(\sum _{i=1}^{n}\pi _{i}P_{i}\right)-\sum _{i=1}^{n}\pi _{i}H(P_{i})\end{aligned}} where

{\begin{aligned}M&:=\sum _{i=1}^{n}\pi _{i}P_{i}\end{aligned}} and $\pi _{1},\ldots ,\pi _{n}$ are weights that are selected for the probability distributions $P_{1},P_{2},\ldots ,P_{n}$ , and $H(P)$ is the Shannon entropy for distribution $P$ . For the two-distribution case described above,

$P_{1}=P,P_{2}=Q,\pi _{1}=\pi _{2}={\frac {1}{2}}.\$ Hence, for those distributions $P,Q$ $JSD=H(M)-{\frac {1}{2}}{\bigg (}H(P)+H(Q){\bigg )}$ ## Bounds

The Jensen–Shannon divergence is bounded by 1 for two probability distributions, given that one uses the base 2 logarithm.

$0\leq {\rm {JSD}}(P\parallel Q)\leq 1$ With this normalization, it is a lower bound on the total variation distance between P and Q:

${\rm {JSD}}(P\parallel Q)\leq {\frac {1}{2}}\|P-Q\|_{1}={\frac {1}{2}}\sum _{\omega \in \Omega }|P(\omega )-Q(\omega )|.$ For log base e, or ln, which is commonly used in statistical thermodynamics, the upper bound is ln(2):

$0\leq {\rm {JSD}}(P\parallel Q)\leq \ln(2)$ A more general bound, the Jensen–Shannon divergence is bounded by $\log _{2}(n)$ for more than two probability distributions, given that one uses the base 2 logarithm.

$0\leq {\rm {JSD}}_{\pi _{1},\ldots ,\pi _{n}}(P_{1},P_{2},\ldots ,P_{n})\leq \log _{2}(n)$ ## Relation to mutual information

The Jensen–Shannon divergence is the mutual information between a random variable $X$ associated to a mixture distribution between $P$ and $Q$ and the binary indicator variable $Z$ that is used to switch between $P$ and $Q$ to produce the mixture. Let $X$ be some abstract function on the underlying set of events that discriminates well between events, and choose the value of $X$ according to $P$ if $Z=0$ and according to $Q$ if $Z=1$ , where $Z$ is equiprobable. That is, we are choosing $X$ according to the probability measure $M=(P+Q)/2$ , and its distribution is the mixture distribution. We compute

{\begin{aligned}I(X;Z)&=H(X)-H(X|Z)\\&=-\sum M\log M+{\frac {1}{2}}\left[\sum P\log P+\sum Q\log Q\right]\\&=-\sum {\frac {P}{2}}\log M-\sum {\frac {Q}{2}}\log M+{\frac {1}{2}}\left[\sum P\log P+\sum Q\log Q\right]\\&={\frac {1}{2}}\sum P\left(\log P-\log M\right)+{\frac {1}{2}}\sum Q\left(\log Q-\log M\right)\\&={\rm {JSD}}(P\parallel Q)\end{aligned}} It follows from the above result that the Jensen–Shannon divergence is bounded by 0 and 1 because mutual information is non-negative and bounded by $H(Z)=1$ .

One can apply the same principle to a joint distribution and the product of its two marginal distribution (in analogy to Kullback–Leibler divergence and mutual information) and to measure how reliably one can decide if a given response comes from the joint distribution or the product distribution—subject to the assumption that these are the only two possibilities.

## Quantum Jensen–Shannon divergence

The generalization of probability distributions on density matrices allows to define quantum Jensen–Shannon divergence (QJSD). It is defined for a set of density matrices $(\rho _{1},\ldots ,\rho _{n})$ and a probability distribution $\pi =(\pi _{1},\ldots ,\pi _{n})$ as

${\rm {QJSD}}(\rho _{1},\ldots ,\rho _{n})=S\left(\sum _{i=1}^{n}\pi _{i}\rho _{i}\right)-\sum _{i=1}^{n}\pi _{i}S(\rho _{i})$ where $S(\rho )$ is the von Neumann entropy of $\rho$ . This quantity was introduced in quantum information theory, where it is called the Holevo information: it gives the upper bound for amount of classical information encoded by the quantum states $(\rho _{1},\ldots ,\rho _{n})$ under the prior distribution $\pi$ (see Holevo's theorem). Quantum Jensen–Shannon divergence for $\pi =\left({\frac {1}{2}},{\frac {1}{2}}\right)$ and two density matrices is a symmetric function, everywhere defined, bounded and equal to zero only if two density matrices are the same. It is a square of a metric for pure states, and it was recently shown that this metric property holds for mixed states as well. The Bures metric is closely related to the quantum JS divergence; it is the quantum analog of the Fisher information metric.

## Generalization

Nielsen introduced the skew K-divergence: $K_{\alpha }(p||q)=\mathrm {KL} (p||(1-\alpha )p+\alpha q)=\int p(x)\log {\frac {p(x)}{(1-\alpha )p(x)+\alpha q(x)}}\mathrm {d} x.$ It follows a one-parametric family of Jensen–Shannon divergences, called the $\alpha$ -Jensen–Shannon divergences: $\mathrm {JS} _{\alpha }(p,q)={\frac {1}{2}}\left(K_{\alpha }(p||q)+K_{\alpha }(q||p)\right)=\mathrm {JS} _{\alpha }(q,p),$ which includes the Jensen–Shannon divergence (for $\alpha ={\frac {1}{2}}$ ) and the half of the Jeffreys divergence (for $\alpha =1$ ).

Another generalization of the Jensen-Shannon divergence consists in considering mixtures with respect to a mean M (like the geometric mean instead of the arithmetic mean). The statistical M-mixture is $M(P,Q)=M(p(x),q(x))/Z_{M}(p,q),$ where $Z_{M}$ is the normalizing factor: $Z_{M}(p,q)=\int M(p(x),q(x))dx.$ Then the generalized Jensen-Shannon divergence is

${\rm {JSD}}_{M}(P\parallel Q)={\frac {1}{2}}D(P\parallel M)+{\frac {1}{2}}D(Q\parallel M)$ where $M=M(P,Q)$ . The geometric Jensen-Shannon divergence between densities of an exponential family is then obtained in a closed-form formula.

## Applications

The Jensen–Shannon divergence has been applied in bioinformatics and genome comparison, in protein surface comparison, in the social sciences, in the quantitative study of history,, fire experiments and in machine learning.